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Effect  of  Point-in-Time  in  Instruction 
on  the  Measurement  of  Achievement 


The  measurement  of  achievement  is  a  considerably  more  complex  problem 
than  the  measurement  of  ability.  Whereas  ability  levels  develop  over  long 
periods  of  time  and  remain  relatively  stable  with  exposure  to  different  en¬ 
vironments,  achievement  levels  characteristically  result  from  exposure  to 
specific  instructional  or  training  environments.  Although  these  instructional 
environments  may  span  relatively  long  periods  of  time,  such  as  the  2-  to  4-year 
periods  required  for  training  in  many  skilled  fields,  instructional  decisions 
are  more  typically  made  on  the  basis  of  instructional  periods  of  several  weeks 
or  a  few  months.  In  the  extreme  case,  as  in  computer-assisted  instruction, 
the  instructional  environment  designed  to  modify  individuals'  achievement  levels 
may  be  as  short  as  a  few  minutes.  Thus,  achievement  is  a  dynamic  variable 
which  may  change  over  very  short  time  intervals. 

The  Pretest. -Tee t  i\ 1 1_ \ iJicjn 

Because  achievement  levels  should  be  sensitive  to  instruction,  it  may  bo 
desirable  to  measure  achievement  (1)  prior  to  instruction,  (2)  at  the  end  (or 
peak)  of  instruction,  and  (3)  some  time  after  the  completion  of  instruction. 

The  measurement  of  achievement  prior  to  instruction  is  accomplished  by  means 
of  pretests,  designed  to  determine  a  student's  level  of  achievement  before  ex¬ 
posure  to  the  instructional  environment.  To  determine  whether  instruction  has 
had  an  effect,  a  pretest-test  paradigm  may  be  used,  in  which  group  or  individ¬ 
ual  gain  scores  are  computed  to  demonstrate  the  effects  of  the  instruction. 

The  pretest-test  paradigm  for  measuring  achievement  has  at  least  two  major 
problems.  First,  both  individual  and  group  change  scores  have  been  shown  to 
be  highly  unreliable  (Cronbach  &  Furby,  1970)  unless  the  pretest  and  peak-of- 
instruction  tost  measurements  are  each  extremely  precise.  Second,  it  is 
necessary  to  administer  two  tests  covering  the  same  material  to  students.  If 
the  same  test  is  administered  prior  to  instruction  and  at  the  peak  of  instruc¬ 
tion,  students  may  not  be  motivated  to  respond  optimally  to  both  tests;  thus, 
precision  of  measurement  on  the  tests  may  be  lowered.  At  the  same  time, 
students  may  remember  test  items  from  the  pretest,  find  the  answers  to  those 
items  in  the  course  materials,  and  then  perform  better  on  the  second  test  than 
they  would  have  performed  had  they  not  seen  the  test  items  prior  to  instruction. 

On  the  other  hand,  if  different  tests  are  used  prior  to  instruction  and 
at  the  peak  of  instruction,  the  serious  problem  of  developing  parallel  tests 
arises.  This  is  compounded  by  the  necessity  to  obtain  very  precise  measure¬ 
ments,  which  may  result  in  pretests  and  peak-of- instruction  tests  measuring 
different  aspects  of  the  achievement  variable  in  the  quest  for  highly  precise 
measurements. 

A  potential  solution  to  these  problems  in  the  pretest-test  paradigm  lies 
in  the  application  of  techniques  of  item  characteristic  curve  (ICC)  theory 


ll.ord  >s  Novlck,  WoSl  tv<  t  In'  measurement  v't  achievement.  Achievement  test 
items  calibrated  usiiiv;  UY  theory  ^e.*.,  Bo|ar,  Weiss,  >s  Kingsbury,  will 

.ill  be  v>t\  tb«‘  same  metric.  I'lnis,  select  ion  ot  UY  cal  ibraicd  items  l  tom  the 
•..ime  item  poo  l  t constitute  pretests  .ui.l  peak  .'t  institution  tests  will,  in 
theory,  eliminate  the  nee.i  loi  the  construction  v't  parallel  tests.  In  addition 
plvictutt  all  the  items  v>n  the  same  metric  bv  using  U‘i‘  item  parameters  will  elitu 
Inate  the  nee.!  tv>  iepe.it  the  same  items  at  the  two  testings,  since  vagaui  in 
theoivl  anv  subset  v't  items  t  rom  the  preval  lhi.it  ed  pv'v'l  will  measure  t  lit'  sumo 
variable  as  anv  olhoi  subset  v't  items.  Thus,  items  toi  pretests  anil  toi  peak 
v't  lust  rue  t  Ion  tests  can  bo  selected  t  rom  the  UY-c.il  ibrut  oil  pv'v’l  on  the  basis 
v't  vontent  cons  tvlerat  ions  result  lug  in  ettectlvelv  paiallel  me.isui  v-ment  s . 

UY  theory  can  also  be  applied  lo  the  pretest-test  paradigm  v't  achievement 
measurement  through  the  use  v't  adapt  ive  test  in.;  tv>  increase  the  precision  v't  th 
achievement  me.isu.  ement  s ,  thus  pv'ssiblv  'crmitliug  the  use  v't  iiivlivivlu.il  v>i 
group  ji.i  in  scores.  Kcscaich  b\  Botai  ami  Weiss  iBclat  .\  Weiss,  l  '>  *  S ;  lu'jai, 
Weiss,  .\  nlalluca.  l‘)'  i  vn  an  achievement  testing  Cv>ute\t  sh.'ws  that  UY -based 
adapt  ive  tests  produce  meusui  ement  s  which  are  Cv'ns  ulerab  l  \  mv'ie  pi  oo  i  so  than 
thv'se  v't  convv'iit  lou.i  l  ach  lovement  tv'st-.,  support  ing  simil.u  findings  in  t  ho 
ability  testing  literature  p'.;.  ,  Me  hi  ivle  .v  Weis-.,  l'>'(>;  Vale,  l  '»  '  > ;  Vale  .\ 

We  ic.,  I  ’  si  . 

Hetv'ie  U'l'  iheoiy  Cvin  be  applied  m  the  pietest  l.".|  p.uadigm  v't  a,  hteiv 
ment  measut  v-ment  ,  however,  It  must  be  vlv'iiii'iist  i  at  ed  that  vlata  obtained  in  this 
paiadtgin  meet  t  he  as  sump  l  tons  v't  t  hi-  t  heoi  \  .  Spec  it  icullv,  suicv'  niv'st  li'i' 
b.isv'vl  t  v'chn  iv|iiv's  iv'v|uiiv*  uu  ivl  miens  lona  1  1 1  \  ,  it  must  bv'  shown  that  both  pietests 
ami  pv'.ik  v't  instruct  tv'ii  tests  ate  os. sent  tally  un  ivi  linens  iv'tia  l  v'l  that  ,  in  gen 
v'l.il,  t  he  vt  i  metis  Iona  l  i  t  \  ot  the  two  t  osi  -.  t -.  t  In'  '..miv'.  In  addition,  it  must 
be  demons  1 1  at  ed  that  the  latent  space  in  t  ho  two  tost  s  does  not  change.  that 
is,  even  t  hough  both  the  pietest  and  t  he  peak  v't  instinct  ion  test  me  uuidt 
mens  tonal  ,  it  t-.  possible  that  they  aie  measuring  ach  t  v'veniv'iit  >'n  vlitteieul 
d  linens  ions .  It  this  t-.  the  case,  1 1  em  pnrumeteis  estimated  at  one  pv'int  in 
inst  rue  t  ion  would  not  bv-  usable  at  the  othot  point  m  institution. 


dust  as  measut ed  nclt  lovoment  i  s  expected  tv'  change  m  lv-v.-l  t  i  v'm  piv't.'-.t 
tv>  pv'.ik  v't  inst  i  net  Ion,  it  t  •.  also  expected  that  it  should  remain  stable  toi 
some  t  title  allot  inst  r  uc  t  ton .  I'lnis  ,  it  is  appropriate  tv'  investigate  whet  hot 
■vusuiod  .iv'hieveiuv'iit  U'vels  detotioruto  v'vv'i  -ihv'i  t  v'i  lor.;  p.-i  iods  v't  t  ime  in 
v'tvlei  tv>  vlv't  el  mini'  tin'  pel  m.inonc  v  v't  tin'  inst  rmt  iv'nal  ot  Iv'vt  vlv'niv'iist  i  at  ed  by 
t  ho  pretest  lest  vl.it  a.  Such  a  dcmoiistt.il  ion  would  require  t  ho  tost  post  tost 
p.uadigm  in  whivh  the  peak  v’t  Instinct  ion  tost  is  followed  at  some  pv'int  in 
t  ime  bv  tin'  admin  i  si  i  at  ion  v't  a  post  test. 

Because  t  he  test  post  lest  paradigm  toi  measuring  constancy  v't  ach  ievement 
may  bv'  Implemented  with  etthoi  the  parallel  tests  approach  ot  the  topoutod 
tests  approach,  it  has  exactly  the  same  problems  a-,  the  pretest  tost  patudigm. 
Similarly,  the  use  ot  WY  tlnviv  ainl  .nlapt  i  y  ,•  test  mg  may  bv'  brought  tv'  bv'.it 
v'n  tlies,'  problems  it  the  pv'.ik  v't  Instruct  ton  and  posttost  vlata  niv'v't  the  i  ,• 
v|u  1 1  v-ment  ■■  v't  these  uppi  ouches .  thus,  simil.u  kimls  v't  vlata  must  be  generated 
lv>  investigate  the  us,-  ,<  t  th,".,-  approaches  in  an  achievement  context. 

Die  use  v't  a. (apt  ive  test  ms  in  the  measurement  v't  ach t ovemett t  whv'thv'i  at 
pretest,  pv'.ik  v't  mst  i  in  t  t.'ii ,  oi  post  test  t.nsos  an  .nlvl  1 1  1,'iia  l  problem  whivh 
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requires  Investigation.  To  realize  the  potential  gains  in  the  measurement  of 
achievement  in  increased  precision  (Bejar,  Weiss,  &  Gialluca,  1977),  higher 
validity  (Bejar  &  Weiss,  1978),  and  shorter  testing  times  (Brown  &  Weiss,  1977) 
adaptive  tests  should  be  administered  by  computer  In  achievement  environ¬ 
ments  with  large  numbers  of  students,  there  may  not  be  sufficient  numbers  of 
adaptive  testing  terminals  so  that  each  student  can  be  tested  at  the  peak  of 
his  or  her  instruction.  Thus,  it  may  be  necessary  for  students  to  have  their 
achievement  measured  at  some  point  beyond  the  peak  of  instruction.  A  similar 
situation  exists  in  self-paced  instructional  environments,  where  students  may 
not  take  achievement  tests  exactly  at  the  peak  of  instruction  due  to  procras¬ 
tination  or  influences  beyond  their  control  (e.g.,  unavailability  of  equipment) 
In  both  cases,  it  is  an  important  question  whether  achievement  measured  after 
the  peak  of  instruction  is  measured  on  the  same  dimension  as  achievement  mea¬ 
sured  at  the  peak  of  instruction. 

Objectives 


The  studies  reported  below  were  designed  to  investigate  several  questions 
-relevant  -to  the  implementation- of  ICC -theory  in  pretest-test  and -test-post test 
paradigms  for  measuring  achievement.  The  data  also  have  some  bearing  on  the 
practical  questions  involved  in  the  use  of  adaptive  testing  in  measuring 
achievement  within  the  realistic  constraints  of  instructional  environments. 


STUDY 


EFLA 77 ONSHIP  OF  TEST  CHAEACTERTST ICS  PEIOR  TO  INSTRUCTION 


AND  AT  PEAK  OF  INSTRUCTION 

This  study  was  designed  to  investigate  two  questions  concerning  test  char¬ 
acteristics  of  a  test  used  to  measure  achievement  at  the  peak  of  instruction 
when  it  was  applied  to  a  population  of  testees  measured  prior  to  instruction: 

1.  Are  ICC  item  parameters  estimated  from  data  obtained  prior  to 
instruction  quantitatively  equivalent  to  parameters  estimated 
at  the  peak  of  instruction?  This  question  is  concerned  with 
whether  the  ICC  metric  maintains  its  interval  properties  during 
the  course  of  instruction. 

2.  Do  tests  used  to  measure  achievement  prior  to  instruction  (i.e., 
pretests)  measure  attributes  from  the  same  latent  space  as  tests 
u 'ed  to  measure  achievement  at  the  peak  of  instruction?  This 
question  is  concerned  with  whether  the  responses  of  the  two  pop¬ 
ulations  (pretest  versus  test)  can  be  described  by  a  common 
latent  space. 

If  the  responses  to  both  of  these  questions  are  affirmative,  the  results  may 
be  taken  as  support  for  the  pretest-test  paradigm.  These  results  would  also 
have  implications  for  the  power  of  the  unidimensional  ICC  model  for  measuring 
achievement  during  the  course  of  instruction.  If  major  differences  are  found 
in  the  characteristics  of  the  tests  used  to  measure  achievement  prior  to  and 
at  the  peak  of  instruction,  the  foundation  of  the  pretest-test  paradigm  for 
measuring  achievement  would  be  weakened  in  many  applications  and  the  use  of  the 
ICC  model  to  measure  growth  in  achievement  would  be  limited. 


• jv  nar  r  The  testing  sessions  that  provided  the  prlor-to- 

Instruetlon  data  were  classroom  examinations  administered  on  the  tlrst  day  of 
class  during  the  tall  academic  quarter  ot  l’>77  to  all  students  attending  class 
tor  Biology  1-011,  General  Biology,  at  the  University  ot  Minnesota.  for  a 
description  ot  the  course  and  testing  procedures,  see  BeJ.tr,  Weiss,  Kings¬ 
bury,  l  >■>  7  7 . )  Data  were  obtained  from  l,Jl>-»  students.  l‘he  tost  administered 
at  this  time  consisted  of  •*  0  multiple-choice  items  samp  1  ed  from  all  ot  t  hi*  7 
content  areas  covered  in  the  course;  these  items  were  taken  from  a  larger  pool 
ot  items  developed  tor  this  course. 

•V.t\  The  peak-ot- lust  rue t ion  test  data  were  obtained  from 

two  sources  so  that  two  different  tvpes  ot  quest  tons  .insBcfmi-,  The. 

_ firsX  question  concerned  whethei  item  parameter  estimates  obtained  t rom  pi  lot¬ 
to- inst  met  ion  testing  were  similar  tv’  est  limit  es  obtained  tor  the  same  items 
at  peak-v't  -  inst  rue  t  ion  testing.  I’eak-of- instruct  ion  parameter  estimates  were 
used  that  were  obtained  from  test  data  supplied  bv  individuals  enrolled  in  the 
same  course  during  five  earlier  academic  quarters,  since  it  would  be  inappro¬ 
priate  tv’  administer  the  same  items  twice  tv'  t  ho  same  individuals.  These  cal¬ 
ibration  samples  averaged  between  700  and  1,000  students.  foi  the  exact 
number  ot  subjects  responding  tv’  each  ot  the  Items  tor  calibration,  see  Kings¬ 
bury  v\  Weiss,  l’>7’>,  p.  do.') 

The  second  quest  ion  concerned  whether  the  tuctoi  structure  underlying 
students'  responses  changed  as  a  function  ot  Instruction.  l'o  answer  this  ques¬ 
tion,  student  roponse  data  were  used  which  were  collected  on  a  sS-item  midquar- 
tv-r  examlnat  ion  administered  •*  weeks  atter  the  pretest  ,  as  one  ot  the  coarse 
requirements,  to  approximately  t he  same  group  ot  students  who  took  t ho  pretest. 
Approximately  1,700  students  completed  the  ’'’-item  midquarter  examination. 

Koch  student  was  required  to  omit  S  items  in  t he  examination;  consequent  1 v , 
data  tor  each  item  were  based  on  about  1,000  students. 


.  •••  **  ’  Kst  Imat  Ion  ot  IvY  item  parameters  tor  the  peak- 

v’t-  inst ruct Ion  data  is  described  in  detail  In  Bejar,  Weiss,  and  Klngsburv 
V.  I  7  ‘  1  .  In  brief,  a  computer  program  developed  bv  Frrv  was  used  tv'  tit 

a  three-paramet er  logistic  ogive  tor  each  item  administered  tv'  the  testoes. 
Items  Wv-re  rejected  bv  the  parameter  est  im.it  ion  program  it  t  hev  tailed  tv’  reach 
certain  minimal  standards  with  respect  to  their  dlscrlminat  Ion  value  ft  and 
lower  asymptote  ft.  Thus,  values  for  the  index  ot  discriminatory  power  ft, 
ft,  and  probability  ot  attaining  a  correct  answer  with  no  knowledge  ot  the 
subject  lot  were  obtained  tor  each  I t cm  that  surpassed  the  minimum  standards. 
Spec  1 f leal lv,  an  item  was  rejected  if  during  the  tirst  stage  ot  t  lie  parameter 
estimation  process  the  value  of  its  .t  parameter  estimate  was  less  than  .SO  or 
the  value  of  its  .*  parameter  estimate  was  greater  than  .  Hf. 

This  procedure  was  applied  separately  to  both  the  pr lor-to- inst ruct Ion 
data  and  the  peak-of- Instruct  Ion  data.  The  results,  tor  each  ot  the  -*0  items 
administered  at  both  points  in  instruct  ion,  were  two  comparable  estimates  ot 
the  111'  parameter  values,  varying  onlv  because  ot  sample  fluctuation  ami  t  lie 
difference  in  the  inst rue t  iona l  level  ot  the  two  groups. 


a. ’  :  r. t*  . **».: r**c* .  1 1  t  ho  latent  space  i%  constant  , 

•  a  • 

t  ho  |Mvami'lt'i  cstim.it  os  Item  the  priei  to  ttist  i  net  Ion  and  t  Ho  peak  el  instruct  i. 
Stroups  should  .litter  no  iik'ii'  t  han  tlu-  rst  (mates  obtained  t  i  em  t  wo  sample-.  .it  t  ti< 
s.imo  level  ot  lust  i  tie  t  tv'ii .  Vo  provide  .1  basis  v't  comparison  tv>i  t  lie  priei  to 
lust  met  It'D  versus  poak-.’t  instruction  cot  i  e  Lit  iv'us  ,  It't'  item  pal  .met  ei  est  l 
m.it  es  well-  computed  t  i  v'tii  two  groups  v't  stiuleuts  m  tin-  same  com  se  vltu  liivt  two 
earlier  nuarteis  win'  iinswei  ovl  .1  compar  .il>  l  o  stioup  v>!  items  .it  t  lie  peak  v’t  liistiu. 
t  tv'ii  it  he  peak  peak  jtroup'  .  i'he  peak  peak  data  wele  part  t.illt  repotted  bel.'lc 
hv  Ih'l.u,  Weiss ,  .uni  k  iiivtsl'in  v  v  l'»  *  *>  ,  win'  reported  tin1  peak  pe.-ik  eert  el.-it  iv'us 
tv'i  tin1  :  .linl  p.n.imetei  estimates  tv'i  1 8  iti'iiis  v'htainevl  l  t  orn  responses  v't 
appi  v>\ iiti.it  e  l  v  '*00  testeos  m  each  saiup  l  v*.  Appendix  Vahle  \  ••tn'ws  item  lunnl'ei  s 
ami  paiametei  est  imates  v't  the  items  usevl  t  v>  invest  i«;at  e  s.-uiip  l  I  vaiiat  i.'ii  in 

parameter  estimates  obtained  t  r.'iii  the  two  groups  at  tin-  peak  ,'t  lust  i  tie  t  ion . 
Alsv>  in  thlri  table  .lie  the  times  v't  admin  is t ra t lou  v't  tin'  items  tv'  tin1  st  intent  s 

lv*  compute  It't'  paiametei  est  tni.it  es  v'bt  a  im-vl  pi  i.'i  to  instinct  ten  with 
t  he  so  obtained  at  tin'  peak  v't  i  ust  i  tie  t  ion  ,  I’eat  sv>u  ptv'iliiet  moment  ooi  t  el. -it  tv'tis 
wete  com'plrt'n.i  between  idem  paiametei  est  imates  obt  a  tue.l  at  the  two  time  pet  teds 
VP  riot  peak  oovrelat  ions'  sepat  at  el  v  toi  the  ,  an.l  •  pavamet  et  s'.  the 

three  eet  i  el.tt  tens  obtained  were  also  computed  in  the  two  samples  which  weie 
at  tin1  peak  et  instruct  ion  tpeak  peak  eei  rel.it  ions'  .  I'ei  each  et  I  he  thiee 
painmeteis,  the  priei  peak  and  peak  peak  eeiielat  lens  loui.l  .lit  1 1*  i  oul\  te 
tbe  extent  that  the  tmltvivluals  di  tiered  m  then  tost  pel  t  ei  matice  when  the 
Stroup  was  tested  priei  to  instruct  ion  e  empaled  with  t  he  1 1  per  to  nuance  when 
tested  at  the  peak  et  i  ns  t  me  t  ion ,  as  retie, 'ted  in  the  UY  item  paramet  ei  s . 
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tested  on  the  tirst  mtd.juattet  examination,  which  was  admtutst  et  ed  latet  in 
the  course.  I'he  items  were  used  to  Investigate  the  tactei  struetuie  pi  lei  te 
inst  i  net  t.'ii . 

Went  v  v'ne  Items  tappiiivt  the  same  content  areas  were  chosen  arbitral  tlv 
Horn  the  tirst  mldiiuartei  cxamiu.it  ion,  which  was  administered  t  o  the  same  tndi 
viduals  <  week-;  attei  the  preti'st.  I'he  students’  responses  te  these 


on 


t  , . ■ 


wore  used  to  exam  i  no  t  ho  tactoi  structure  underlying  portorm.ineo  at  the  peak 
ot  Instruction.  Items  administered  at  the  pretest  and  at  the  t irst  midquarter 
were  sampled  item  the  same  content  areas,  hut  different  items  were  used  at  t he 
t wo  points  in  time. 

For  each  ot  these  groups  oi  items,  the  same  procedure  was  followed  to  ob¬ 
tain  the  final  tactor  structure.  first  ,  all  student  responses  were  scored  "O’* 
it  incot  root  or  "1"  it  coi  rcct  .  Soeond,  those  recoded  responses  were  used  t  o 
obtain  tetrachoric  correlations  among  the  items  through  the  PFl'KAOHORU'  sub¬ 
routine  in  the  Statistical  Package  tor  the  Social  Sc  lonoo..  t.Nio,  Hull,  Jenkins, 
Ste  inbrenner ,  >s  bent,  1'* '01.  I’he  two  resultant  correlation  matrices  were  then 
tactot  analv.ed  using  the  F.U'l’OR  subroutine  from  the  same  statistical  package, 
the  tinal  tactoi  solutions  were  obtained  using  a  principal  axis  solution;  the 
initial  conununalitv  estimates  were  the  squared  multiple  correlations  of  each 
variable  with  all  the  othot  variables.  The  tactor  solutions,  which  were  arbi 
trarilv  limited  t>'  live  tactors,  were  iterated  until  tin-  ditterences  in  suc¬ 
cessive  communal l t v  estimates  were  negligible.  This  procedure  provided  the 
t  i  no  l  so  lull  oils  . 

The  two  tinal  tactoi  solutions  were  then  compared  for  similarities  and 
ditterences  in  terms  ot  the  number  ot  salient  tactors,  the  strength  ot  each 
tactot  ,  and  the  amount  ot  variance  in  the  item  intercorrel.it  ions  accounted  for 
b\  the  tactoi  solutions.  lo  the  extent  that  observed  ditterences  between  the 
two  solutions  were  minor,  it  could  ho  inferred  that  tin-  underlying  factors 
contributing  to  testee  responses  were  the  same  print  t o  instruction  and  at  the 
pea  k  ot  instruction.  Vo  the  extent  that  major  J  i  sc  rop.inc  i  es  were  observed  be¬ 
tween  the  so lut ions ,  it  could  bo  Inferred  that  ditterences  existed  in  the 
structure  ot  achievement  at  the  two  points  in  instruction. 


Parameter  estimates  tor  each  ot  the  *0  items  administered  in  the  prlor-to- 
instruction  achievement  measure  are  shown  in  Pablo  1,  along  with  parameter 
estimates  tor  the  same  items  obtained  t rom  groups  ot  testees  at  the  peak  ot 
instruction.  From  this  table  it  can  be  seen  that  !■•  ot  the  items  tailed  to 
meet  the  minimal  standards  ot  the  estimation  procedure  when  administered  prior 
to  instruction  and  '  items  failed  when  administered  at  the  peak  ot  instruction. 
Four  items  were  rejected  in  both  instances.  .\ttor  all  ot  the  rejected  items 
were  removed  from  cons ivlerat  ion ,  .'■>  items  remained  lor  the  oorrol.it  ional  aualv 
sis. 


Pb  •  bivariate  plots  ot  .  .  and  parametet  estimates  t rom  the  two  cal- 

ibrat  io  are  shown  in  Figures  l,  ,  and  ',  respectively.  It  can  be  seen  from 
these  tt  .res  that  the  relationships  between  the  sample  estimates  ot  the  para¬ 
meter  values  were  weak,  at  best.  Figure  l  shews  a  eorrelat  ton  ot  - .  1 for  the 
parameter,  indieat ing  a  slight  tendency  for  high  values  of  at  peak  ot  in¬ 
struction  to  be  associated  with  low  values  prior  to  instruction. 

Phe  eorrelat  ion  ot  •  .b*  for  the  .  paiametei  data  show  a  tendency  tot  high 

values  ot  prior  to  instruction  to  be  associated  with  high  values  at  the  peak 
ot  instruct  ion.  However,  almost  all  the  data  points  in  Figure  are  below  the 

be  more  diffienlt  prior  to 
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Table  1 

Parameter  Estimates  of  Items  Calibrated 
Prior  to  Instruction  anJ  at  the  Peak  ot  Instruction 


Item 

Number 

Prior 

to  Instruction 

Peak 

ot  Instruct  ion 

v? 

b 

a 

b 

30  35 

1.21 

2.53 

.  28 

.90 

.  68 

.  28 

32s  1 

1 . 89 

2.87 

.  48 

.91 

2.09 

.  17 

38  Ih 

1.72 

2b 

.51 

— 

— 

— 

4013 

1 . 02 

.  79 

.  38 

1.76 

-1.88 

.10 

1809 

1.12 

.70 

.  15 

1.27 

-.61 

.53 

4010 

1.25 

.00 

...  1 

.  88 

-1.82 

.23 

3817 

— 

— 

— 

— 

— 

— 

380  i 

— 

— 

— 

— 

— 

— 

3210 

— 

— 

— 

1 . 04 

-1.22 

.40 

38  3 ' 

1.10 

1.13 

.  29 

1.09 

-1 .59 

.25 

3235 

1.17 

1.38 

.49 

1.15 

-1.40 

.  28 

3808 

— 

— 

— 

.  99 

-1.00 

.  30 

4033 

— 

— 

— 

.90 

2.23 

.  38 

3812 

.  ss 

1.92 

.  33 

.  82 

-.o3 

.  13 

3424 

— 

— 

— 

— 

— 

— 

3821 

— 

— 

— 

.90 

- .  o  j 

.43 

3244 

1 . 00 

2.  52 

r  -y 

.  *4  / 

l .  15 

- .  44 

.  23 

301  3 

.91 

1.18 

.  38 

1 . 00 

-.97 

.  39 

300  5 

1  .  55 

.03 

.  4  b 

1  .  17 

-  1 . 00 

.  14 

3909 

— 

— 

— 

1 .  3s 

.77 

.  18 

J922 

.  76 

2.3  3 

.  J8 

.  b4 

-.  2b 

.  30 

3s  15 

— 

— 

— 

.  85 

-.90 

.41 

3428 

— 

— 

— 

.90 

-1.56 

.40 

306  7 

.98 

1.15 

.  13 

1.07 

-.76 

.21 

3272 

.83 

1 ...  5 

.40 

1 .  Oo 

-.81 

.  37 

3908 

— 

— 

— 

1.15 

.07 

.  11 

3435 

1.85 

1 . 4  3 

.  39 

.83 

-.0  1 

.  *4  J 

4005 

— 

— 

— 

— 

— 

— 

342o 

1.03 

2.68 

•  4 

.  68 

.07 

3031 

.  73 

2.54 

•  2s 

1.47 

-.  33 

.  19 

4006 

1.01 

2.53 

.47 

.84 

-.  59 

.  lo 

30  b  9 

.  83 

1 . 11 

.45 

.88 

-.01 

.  ^8 

3211 

— 

— 

— 

.  88 

.01 

.  1 3 

3905 

— 

— 

— 

.98 

.  35 

.20 

4015 

.  7  b 

2 . 26 

.  11 

2.0  3 

-1 .02 

.12 

140  i 

.93 

1 .  38 

.  33 

.94 

.18 

.  19 

3000 

3.0o 

2 .  46 

.21 

1.24 

.52 

.  36 

344  5 

.73 

2.50 

.  19 

1.19 

.  34 

3218 

2 .  4  4 

2.18 

.30 

.82 

.  58 

.12 

4001 

1.44 

2.49 

.  29 

1 .47 

- 1  .  1  S 

.13 

Not.,'. 

Miss ing 

values  indii 

ate 

that  item 

was  re  1 oe 

ted 

by  the  item  calibration  procedure. 


instruction  than  at  the  peak  ol  instruct  ion.  The  data  in  Table  1  indicate 
that  prior  to  instruction  only  one  ot  t  lie  item  ;•  values  was  negative  fan  easy 
item),  but  at  peak  of  instruction  more  than  half  the  items  had  negative  b  values. 
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test  inn  pool.  Using  the  3  * -transformation ,  85%  con  1 i doin'  e  intervals  were  com¬ 
puted  for  each  of  the  prior/peak  correlations  with  the  following  results: 

1.  For  the  Index  of  item  discrimination,  the  lower  limit  of  the  85% 
confidence  interval  around  the  prior/peak  correlation  was  -.50;  the 
upper  limit  was  .JO.  The  peak/peak  correlation  for  this  parameter 
was  .  <i  J ,  which  was  beyond  the  bounds  of  the  confidence  interval. 

2.  For  the  index  ot  item  difficulty,  h ,  the  limits  ol  the  8' >X  confidence 
interval  were  .  il  and  .81.  The  peak/peak  correlat  ion  was  .80,  which 
was  beyond  the  limits  of  the  confidence  interval. 

1.  For  the  index  of  the  lower  asymptote  ol  the  ICO,  ,  the  limits  of 
the  85%  confidence  interval  were  -.J7  and  .55.  The  peak/peak  cor¬ 
relation,  .41,  fell  w it h in  l he  bounds  ot  the  coni i deuce  interval. 

It  should  be  noted  that  t  lie  correlation  between  the  estimates  of 
the  , ’-parameter  was  quite  low,  even  in  comparable  samples,  account - 
Ing  for  less  than  .10".  common  variance. 

F<U'ti'V  Ft)'; „•?;,■  tv 

Item  Intercorrelation  matrices  for  t  lie  two  71-item  subsets  are  in  Appendix 
Table  H.  Final  values  ot  the  communa  1  1 1  v  estimates  are  shown  in  fable  t.  The 


Tab  1 e  3 

Final  I’.ommuna  I  i  t  v  list  (mates 
Used  in  Factor  Analysis  ol 


Pretest  and  First  Midqu.ntoi  Fsamination 


1  t  cm 

Pi  et  est 

First 

Midquart  or 

1 

.  341 

.  828 

.024 

.  1  (i(i 

1 

.1‘>1 

.2  11 

4 

.4  7b 

.140 

5 

.127 

.  860 

b 

.052 

.411 

7 

.  188 

.  1  34 

8 

.  178 

.218 

8 

.Od  8 

.278 

10 

.0/8 

.48  4 

1  1 

.24  2 

.  145 

1.’ 

.  08  1 

.  2  40 

1  J 

.  708 

.188 

14 

.OU 

.441 

1  5 

.  22  4 

.  146 

lb 

.  178 

.487 

17 

.110 

.182 

18 

.  2b 4 

.  1  88 

18 

.082 

.  44  1 

20 

.0  14 

.  165 

21 

.171 

.114 

i 
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mean  communal l ty  estimate  for  the  pretest  Items  prior  to  Instruction  was  .20; 
the  mean  communal  tty  estimate  for  the  first  mtdquartov  Items  at  the  peak  ot 
instruction  was  .14.  About  one-third  ol  the  variance  lor  the  average  t est  Item 
at  the  peak  of  Instruction  was  accounted  for  by  the  common  factors,  compared 
to  one-fifth  of  the  variance  for  the  average  Item  prior  to  instruct  inn.  Con¬ 
sequently,  there  was  more  unique  variance  In  l  hi'  item;  administered  pi  ior  to  In 
st ruction  than  in  the  items  administered  at  the  peak  ot  instruction. 

Eigenvalues  lor  the  linal  factor  solutions  are  presented  graphically  in 
Figure  4  (factor  loadings  are  in  Appendix  Table  Cl .  It  can  hi'  seen  that  sub¬ 
sequent  to  the  first  factor,  the  eigenvalues  for  each  factor  were  quite  simi¬ 
lar.  For  the  first  factor,  though,  the  eigenvalue  obtained  from  the  items 
administered  at  the  peak  of  instruction  (4.b7)  was  move  than  twice  the  value 
obtained  from  those  administered  prioi  to  instruction  (2.04).  Thus,  although 
both  sets  of  items  had  a  dominant  first  (actor,  in  the  peak-o I -l ra i u ing  data  a 
stronger  first  (actor  was  evident:  in  the  prior-lo-train ing  data  a  weak  single 
factor  accounted  lor  achievement. 


Figure  4 

Eigenvalues  of  Factors  Obtained  Prior  to  Instruct  ion  (Pretest) 
and  at  Peak  ol  Instruction  (First  Midquarter  Examination) 


•  Pr  i  or  l  o  1  ust  met  ion 
-  At  Peak  el  Inst  met  ion 


Tlu'  correlational  analysis  indicated  that  the  KY  pat  anui  it  s  obtained  t  rum 
t wo  samples  differing;  in  instruct  it'll  were  nune  discrepant  than  was  to  he  ex 
peeted  trout  observing  correlations  among  parameters  obtained  I  rout  samples  at 
the  same  point  in  instruction.  Kurt  her,  tor  two  ol  the  ltY  patamet ers — d ts- 
cr  Intinat  ion  (.;)  and  difficulty  (  I  the  prioi  peak  cot  to  1  a  l  ions  were  I  omul  t  o 
be  s  Ign  i  t  leant  1  y  smaller  than  the  poak/peak  corrcl.it  Ions.  These  findings  implv 
that  the  latent  space  under l  v  lug  tester  responses  chanp.es  enough  between  the 
belt  inn  ing  and  the  peak  ot  instruct  ion  so  that  the  lest  responses  cannot  be  do 
scribed  bv  a  single  latent  coni  inuum;  items  change,  not  on  1 1  with  respect  to 
their  ability  to  d t t Cerent  lat e  testees  at  dttterent  levels  ol  tin-  trait  con¬ 
tinuum  (discrimln.it  ion! ,  but  a  I  so  with  respect  to  t  lie  i  t  relative  difficulties. 
The  findings  imply  that  in  order  to  desci  I  he  test  item  response s  obtained  both 
prior  to  and  at  the  peak  ol  instruction  with  a  sinp.li>  latent  trait  model,  the 
un  id  (mens  iono  I  model  that  was  considered  adequate  t  o  describe  pe  r  I  orm.ince  at 
the  peak  ot  instruction  would  need  t o  be  expanded  to  a  mult  id imens i ona 1  modi' I 
developed  I  rum  both  sets  ol  data. 

This  conclusion  is  supported  by  the  comparative  tactor  analyses.  These 
analyses  showed  a  major  dlftoronce  in  t he  strength  ol  the  first  taotoi  undei 
lying  testi'e  respv'nses  in  the  few  weeks  ot  i  ns  t  met  i  on  between  the  ptetcsi  and 
the  peak  ol  instruction.  Students'  tost  item  responses  were  not  .  t o  as  great 
an  extent,  related  I o  the  tirst  tactor  during  t ho  protest.  Again,  the  imp  I i 
cation  is  that  students  were  no l  responding  t o  the  same  ini luences  i c  I  ho  same 
degree  on  i  he  pretest  and  on  the  m  Idiptor  t  or  cxumin.it  ion  . 

These  findings,  taken  as  a  whole,  imply  that  the  pretest  -lost  paradigm 
may  be  invalid  in  some  instances  simply  because  t he  tests  might  not  ho  tapping 
the  same  underlying  achievement  variable.  This  muv  account  ,  in  part,  lot  the 
lack  ot  reliability  ot  change  scores  reported  in  many  studies  (e.g.,  Tronbach 
ik  Furby,  ll)70;  Harris,  I'lhll.  li  is  suggested  that  the  undei  lying  l.ictoi 
structures  in  testi'e  responses  be  explored  whenever  possible  betore  importance 
is  attributed  to  any  pretest-test  measure  ol  change. 


Tlu'  stability  ot  achievement  estimates  measured  altei  the  peak  ot  instruc¬ 
tion  is  important  for  al  least  two  reasons.  First,  it  i m  lrei)uently  necessary 
to  measure  the  achievement  levels  ot  some  individuals  at  a  ditloront  time  than 
others.  Students  occasionally  miss  examinations  lor  a  variety  ot  reasons  and, 
there  tore,  take  the  exam i nut  ion  at  a  point  which  may  not  be  at  the  it  peak  ol 
instruction.  Where  tests  are  given  bv  oomput  ers  (e.g.,  as  in  adaptive  testing! 
there  may  not  be  suttlcient  terminal  equipment  available  t o  test  all  student'' 
Immediately  at  the  peak  of  instruction.  It  is  thus  important  to  determine 
whether  the  passage  ol  time  alter  the  completion  ot  instruction  at  feet s  achieve 
ment  level  estimates. 

Measuring  an  individual  alter  the  peak  ot  instruction  i s  a  I  so  a  common 
problem  in  research  studies  attempting  t o  measure  retention.  Similar  t o 
the  pretest -test  paradigm,  the  tost  posttest  paradigm  used  to  measure  rotou 


t ion  assumes  that  the  same  achievement  variable  is  being  measured  and  that 
the  passage  ot  time  does  not  change  the  nature  ot  achievement.  Thus,  it  is 
again  relevant  to  determine  whether  achievement  level  estimates  obtained  after 
the  peak  of  instruction  are  systematically  related  to  those  obtained  at  the 
peak  of  instruction. 

It  it  is  hvpot  lies ized  that  performance  ot  Individuals  tested  some  time 
after  the  peak  ot  instruction  is  a  function  ol  the  same  latent  space  that  In- 
t luences  performance  at  the  peak  ol  instruction,  several  outcomes  would  be 
expected  when  individuals'  achievement  levels  estimated  from  tests  given  at 
different  times  are  compared.  It  the  unid imens Iona  1  latent  space  remains  stat¬ 
ic  with  the  passage  of  time,  achievement  estimates  t or  individuals  measured  at 
different  points  in  time  atter  the  peak  ot  instruction  should  differ  only  as 
a  linear  function  ot  the  time  ot  testing  atter  the  peak  of  Instruction.  For  in¬ 
stance,  it  a  group  ol  individuals  passed  t hrough  a  particular  iust ruct ional 
sequence  were  tested  at  the  end  ot  instruction,  and  then  at  a  later  date 
were  brought  back  to  In  tested  again  on  the  same  material,  a  single  linear 
t rails  format  ion  would  be  expected  to  equate  each  individual's  scores  on  the  two 
tests  it  tlte  same  un  id  imens  Iona  1  trait  space  was  in  operation  at  bot  It  times  of 
testing.  This  would  occur  because  t he  metric  underlying  the  trait  space  would 
have  retained  its  interval  properties  (l.ord  &  Novlck,  IdnS)  with  the  passage 
ol  time  and  no  tint  her  instruction  would  have  occurred  that  might  change  the 
ordering  of  the  individuals  in  terms  ot  achievement  level. 

Further,  it  the  same  group  were  brought  back  tor  additional  tests  at  lat¬ 
er  dales,  a  linear  trend  should  be  t omul  in  the  comparison  ot  anv  t wo  testing 
periods,  provided  that  the  latent  space  did  not  change.  If  the  latent  space 
did  vary,  a  linear  tolat  ion  between  the  two  measures  ot  achievement  level  would 
not  be  expected.  I’lius,  it  a  1  ineat  relationship  is  not  observed  between  scores 
obtained  at  and  after  the  peak  ot  instruction,  the  conclusion  can  be  drawn  that 
l wo  different  traits  were  being  evaluated  at  the  two  testing  times. 

This  studv  was  concerned  with  determining  whether  the  uiiidimettsiou.il 
space  defining  achievement  at  the  peak  ot  Inst ruct ion  was  sut fie  lent  to  describe 
achievement  atter  the  peak  ot  Instruction. 


Testing  at  the  peak  ot  instruction  was  a  requirement  for  students  enrolled 
in  the  same  undergraduate  survey  course  in  biology  as  in  Studv  1.  The  peak- 
ot - Inst ruct  ion  tost  data  were  t  tom  the  required  first  midquarter  examinations 
administered  in  a  J-dav  period  to  all  students  enrolled  in  the  course  in  the 
tall  academic  quarter  ot  ll)76  and  in  the  winter  quarter  of  l'*77. 

Testing  atter  the  peak  ot  instruction  was  implemented  bv  the  Computerized 
Adaptive  Testing  Project  using  volunteers  t rom  the  same  biology  classes. 

These  volunteers  were  given  extra  points  toward  their  final  course  grade  for 
participating  in  the  research  and  were  told  that  their  level  ot  performance  on 
the  computer-administered  biology  test  would  have  no  ettect  on  their  final 
grades.  This  testing  began  on  the  day  following  the  midquarter  examtnat  ion 
and  continued  for  approximately  l  month. 


Peak-of-instruction  testing  (the  required  first  midquarter  examination) 
consisted  of  the  conventional  paper-and-pencil  administration  of  5  multiple- 
choice  questions  concerning  the  first  three  content  areas  in  the  course — 
"Chemistry ,"  "The  Cell,"  and  "Energy."  (For  a  more  complete  discussion  of  the 
course  content  and  testing  procedure,  see  Bejar,  Weiss,  ,s  Kingsbury,  1977.) 

The  after-peak-of-instruction  test  (the  second,  voluntary  test)  was  a 
computer-administered  stradaptive  test  (Weiss,  1973)  consisting  of  a  maximum 
ot  50  individually  selected  items  chosen  from  the  same  item  pool  that  was  used 
to  construct  the  required  midquarter,  including  the  same  three  content  areas. 
(For  a  complete  description  ol  this  testing  procedure,  see  Bejar,  Weiss,  & 
Clalluca,  1977).  The  data  used  for  this  study  were  from  251  students  from 
the  fall  quarter  testing  for  whom  achievement  estimates  from  both  tests  and 
the  date  ot  the  later  test  were  available. 

Since  the  adaptive  and  conventional  tests  were  selected  from  the  same  con¬ 
tent  area  pools,  these  two  tests  should  have  measured  the  same  underlying  di¬ 
mension  it  the  passage  of  time  did  not  affect  the  latent  space;  and  although 
differences  in  the  precision  ot  measurement  between  the  two  testing  procedures 
were  present  (Bejar,  Weiss,  &  dlalluca,  1977),  they  should  not  have  affected 
the  outcome  ot  tin1  present  study. 


Peak-of- Inst  met  ton  achievement  level  est  imates  were  obt  ained  by  scoring 
the  students’  midquarter  item  response  data  with  the  scoring  program  L1NDSC0 
(Bejar  &  Weiss,  1979),  wh  ich  is  designed  to  score  conventional  tests  using  item 
character  1st  ic  curve  models.  At t or-peak-ot -inst ruct Ion  achievement  estimates 
were  obtained  by  scoring  the  stradaptive  response  vectors  with  the  program 
AOADSOO  (Bejar  ,5  Weiss,  1979),  which  is  designed  to  score  adaptive  tests  with 
item  character ist ic  curve  models.  Since  the  maxi mum- l ike l (hood  logistic  scor¬ 
ing  mot  hod  list'd  in  both  programs  is  the  same,  the  achievement  level  estimates 
obtained  from  the  two  programs  are  direct Iv  comparable. 


To  determine  whether  the  same  latent  space  was  operative  at  ter  instruction 
that  was  operative  at  the  peak  ot  instruction,  individuals’  achievement  levels 
at  the  peak  of  instruction  were  regressed  on  their  achievement  estimates  alter 
the  peak  of  instruction.  Since  the  later  testing  occurred  over  a  period  ot  a 
mouth,  it  was  possible  to  analyze  t hi'  effect  ot  the  passage  ot  time  on  the  re¬ 
lationship  between  achievement  estimates.  Since  at t er-peak-ot - inst ruct ion 
testing  occurred  only  on  weekdays,  the  weekends  served  as  natural  break  points 
to  divide  the  total  group  of  students  into  four  subgroups,  each  ot  which  was 
tested  during  a  different  week  in  the  month  following  the  peak  ot  instruction. 
Table  s  shows  the  total  number  of  students  tested  each  quarter,  as  well  as  the 
number  tested  in  each  week  following  the  first  midquarter  examination. 

It  the  time  of  testing  after  peak  ot  instruction  at  torts  the  latent  space 
underlying  testee  responses,  this  eftect  mav  be  studied  by  examiu.it  ion  ot  the 
regression  lines  of  peak  testing  on  later  testing  using  data  t rom  each  ot  the 
■<  weeks  of  testing.  To  t  hi'  extent  that  these  regressions  are  parallel  (i.e., 
exhibit  no  Interaction  between  achievement  level  at  the  peak  ot  instruction 
and  the  time  ot  af t er-peak-of- instruct  ton  testing)  and  exhibit  stable  linear 
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It  tho  value  ot  the  statistic  In  Equation  1  Is  statistically  significant, 
the  Individual  regression  lines  are  different  in  some  respect  from  t  ire  overall 
regression  line;  it  is  then  appropriate  to  test  directly  whether  the  Individual 
regression  lines  differ  significantly  in  slope.  This  may  again  he  done  through 
the  use  of  Equation  l.  In  this  instance,  the  restricted  model  becomes  a  single 
regression  equation  predicting  individuals'  later  achievement  estimates  from 
their  achievement  estimates  at  the  peak  of  Instruction  and  from  the  week  of  the 
later  testing.  The  full  model  uses  these  two  predictors  and  adds  the  inter¬ 
action  of  the  two  predictors  to  the  model.  A  significant  value  for  the  sta¬ 
tistic  Indicates  a  significant  inter tct  ion  between  the'  predictor  variables, 
indicating  that  the  Individual  regression  lines  are  not  parallel.  A  signifi¬ 
cant  result  from  this  analysis  would  indicate  a  change  In  the  latent  space. 

These  analyses  were  implemented  tor  both  the  fall  quartet  testing  group 
and  the  winter  quarter  testing  group  in  order  to  examine  the  stability  of  t ho 
results  across  independent  groups. 

For  each  week  ot  testing  tol  lowing  the  peak 
of  instruction,  it  was  desired  to  determine  whether  a  linear  trend  existed  and 
was  sufficient  to  describe  the  prediction  ot  the  a 1 1 er-peak-ot - Inst  rue t ion 
achievement  estimate  from  the  achievement  estimate  obtained  at  the  peak  ot  in¬ 
struction.  This  was  operat iona l i cod  bv  fitting  a  fourth-degree  polynomial 
regression  equat Ion  t o  t hi-  data  tot  each  week  ot  testing  and  separately  deter¬ 
mining  the  significance  ot  each  ot  t ho  terms  in  the  equations.  To  the  extent 
that  these  regression  equal  ions  exhibited  an  increasing  trend  toward  curvi¬ 
linear  lty  as  the  time  between  testings  Increased,  it  could  be  interred  that 
the  latent  space  was  changing  with  time,  causing  a  disruption  in  t ho  interval 
properties  ot  the  original  metric.  It  no  such  trend  was  observed,  it  could  be 
concluded  that  tho  latent  space  remained  stable  as  a  function  ot  time. 

Regression  equat ions  were  obtained  t  torn  t ho  REGRESSION  subprogram  in  the 
Statistical  Package  tor  the  Social  Sciences  iNto.  Hull,  .tonkins.  St einbrenner , 

&  Bent,  ll)70) .  Slmilat  to  the  previous  analysis,  this  analysis  was  imple¬ 
mented  tor  both  tall  and  winter  quarters  t o  permit  replication  ot  the  results 
in  independent  groups. 
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Figure  S  shows  individual  regression  lines  obtained  t rom  each  week  ot 
testing  following  the  peak  ot  instruction  in  t ho  tall  quartet  v Figure  'a'  and 
the  winter  quarter  fFigure  'hi ,  as  well  as  the  restricted  regression  1  Inc 
across  weeks.  Table  *>  shows  the  sum  ot  squares  due  to  ertot  and  other  descrip¬ 
tive  statistics  for  each  ot  the  regression  l tin's  shown  in  Figure  S . 

I'slng  these  sums  ot  squares,  the  test  for  coincidence  ot  regression  in 
Equation  l  resulted  in  an  -value  ot  1.48  tot  the  tall  quartet  data.  I'his 
value,  with  t>  and  .'s'*  degrees  ot  freedom,  had  a  probability  ot  occurrence  bv 
random  fluctuation  of  .  HKfc.JS.  For  the  winter  quarter  data,  the  observed 
-value  was  .dll,  with  h  and  177  degrees  ot  freedom .  The  probability  ot  obtain¬ 
ing  an  f'-value  at  least  this  extreme  bv  random  fluctuation  was 
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Tab  l  e  5 

Number  ot  Subjects,  Sum  of  Squares  Hue  to 
Regression,  and  Sum  ot  Squares  Due  to  Krror  tor 
Week-by-Week  Regressions  and  for  Common  Regressions, 
_ for  Fall  and  Winter  Quarters _ 


Quarter 
and  Week 

.V 

_ _ Sum  o  f 

Error 

St^ua i  os 
Repress i on 

Fall  Quarter 

Week  1 

54 

2 1 . 88 

2  3.  7 1* 

Week  2 

8  1 

33.50 

2  3.52 

Week  3 

8  7 

si .  3: 

3b.  70 

Week  4 

_'4 

l  2 .  54 

18.11 

Common 

253 

113.24 

103. 2b 

Winter  Quarter 

Week  l 

54 

20.77 

13.  32 

Week  2 

40 

58 . 34 

)d 

Week  1 

35 

12.8) 

20.75 

Week  A 

0 

1.  10 

.78 

Common 

1 8 > 

9  ' .  88 

7b.  28 

Neither  obtained  -value  was  large  enough  to  |ustitv  rejecting  the  null 
hypothesis.  This  finding  Implies  that  the  individual  regression  lines  ob¬ 
tained  by  taking  the  time  lapse  after  the  peak  ot  instruction  into  account  pre¬ 
dicted  later  scores  no  better  than  the  single  overall  regression  line.  Since 
the  lines  did  not  differ  significantly,  it  was  not  necessarv  to  test  tor  par¬ 
allelism  among  individual  regression  lines. 


Table  ti  shows  the  significance  ot  each  polynomial  term  • 01  the  separate 
regression  equations  from  each  week  ot  testing  following  the  peak  ot  instruc¬ 
tion,  for  fall  and  winter  quarters.  This  table  shows  that  during  the  fall 
quarter,  the  linear  term  in  the  regression  equation  was  significant,  with  a 
value  of  . 00 1  in  every  week  ot  testing.  lit  onlv  one  instance  was  any  other 
term's  contribution  to  prediction  significant  at  any  reasonable  significance 
level  (i.e.,  ."  .05):  The  quadratic  term  in  the  regression  equation  calculated 
from  the  fourth  week  ot  testing  showed  a  significant  cent r ibut  ion . 

For  the  winter  quarter  the  linear  trend  was  again  significant  at  the  .001 
level  for  the  first  l  weeks  of  testing.  The  fourth  week  ot  testing  resulted 
in  the  only  nonsignificant  linear  trend  in  either  quarter;  this  was  probably 
due  to  the  fact  that  the  regression  equation  tor  t ho  fourth  week  was  based  on 
only  six  students.  It  can  also  be  seen  that  the  quadratic  term  (the  square  ot 
the  peak  achievement  level  estimate)  in  the  regression  equations  obtained  t ot 
each  of  the  first  2  weeks  of  testing  was  a  significant  l.s.051  predictor  ot 
later  performance  for  the  winter  quarter  data.  This  trend  was  not  evident  in 
the  third  week  ot  testing.  Thus,  a  statistically  significant  quadrat  Ic  trend 
was  observed  in  the  winter  quarter  data,  but  this  trend  did  not  increase  as  t  lie 
time  between  testings  Increased.  No  other  high  order  term  contributed  signifi¬ 
cantly  to  the  prediction  ot  later  achievement  level  in  either  the  fall  ot  win¬ 
ter  quarter  data. 
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Table  (' 

Statistical  Significance  l.ovcl  of  Faoh  I'erm  of  the  Fourth  Degree 
Polynomial  Regression  Fquat  ion  Predicting  l.atet 


Performance  from  Performance  at  the  Peak  ot 
Instruction  tor  Fall  and  Winter  Quarters 


Quarto r 
and  Wook 

Linear 

Quadrat ic 

Cub  ic 

Quart ic 

Fall  Quartet 
Week  l 

54 

s.001 

.  158 

.  92  7 

.  o0  t 

Week  2 

83 

. <.001 

.  >88 

.  32 b 

.  1  3s 

Wook  3 

87 

<.001 

.  5  1  0 

.  525 

.SIS 

Wook  * 

JO 

:  v  .001 

.0-.  2 

.  7s 8 

.  s02 

Win tor  Quart t 
Wo  ok  l 

r 

5** 

t's.OOl 

.010 

,o7l 

.  oOO 

Wook  3 

on 

•s.OOl 

.018 

.  8  t  3 

.  <80 

Wo  ok  ' 

:  s . 00 l 

.  .'o0 

.  54  l 

.207 

W  o  o  k  •* 

0 

.  lt>7 

>1 

‘Results  not 

report od 

due  t  o  sm 

ill  .V. 

Since  the  regression  lines  obtained  in  the  different  weeks  ot  after-peak- 
o t - inst rue t ion  testing  did  not  difter  signit  leant Iv  t tom  one  another,  the  re¬ 
sults  ot  these  analyses  did  not  support  the  hypothesis  that  the  achievement 


metric  changed 

as  a  t  uitv' t  ion  o  t 

t  ho 

t  line 

l a  pse  bo t  ween 

t  ho 

peak 

ot  inst  met  ion 

and  later  acltic 

■voment  tost  ing. 

The 

data 

indicate  that 

t  ho 

t  ra  i  t 

space  was 

stable  t>>  the  l 

imit  ot  the  power 

of 

this 

analysis.  It 

ran 

a  1  so 

be  soon  t  rom 

the  data  fsee  Figure  '1  that  student  achievement  measured  after  the  peak  ot 
instruction  was  shitted  in  a  linear  manner  from  that  measured  at  the  peak  ot 
instruction;  this  is  indicated  bv  the  nonzero  intercepts  and  nonunit  slopes  ot 
'he  overall  regression  lines  t tom  both  quarters.  The  difference  mav  be  due  to 
a  lack  ot  motivation  ot  prepar.it  ion  for  t  he  at  t  er-peak-ot  -  inst  rue  t  ion  tests, 
since  that  testing  was  voluntary  and  the  scores  on  that  test  had  no  effect  on 
students'  course  grades.  In  both  quarters,  howevet  ,  there  was  no  evidence  ot 
any  change  in  the  latent  space  with  the  passage  ot  time. 

rhe  polynomial  trend  analysis  indicated  that  in  each  week  ot  testing  tol- 
lowlng  the  peak  ot  instruction,  the  maximum- l ikel ihood  estimates  ot  achieve¬ 
ment  level  at  the  peak  ot  instruction  were  a  significant  linear  predictor  ot  la¬ 
ter  achievement  levels.  This  tindlng  was  consistent  across  academic  •quartets, 
lor  the  tall  quarter  the  quad r a t to  trend  was  significant  only  in  the  t  Inal 
week  ot  testing.  For  t he  winter  quarter  the  quadrat ic  trend  was  a  significant 
predictor  in  the  first  2  weeks  ot  testing,  but  not  in  t  he  third  week.  These 
inconsistent  findings  imply  that  the  quadrat ic  trend  observed  mav  be  a  sample 
art  itact . 

The  results  trout  this  analysis  indicate  that  the  only  polynomial  trend 
that  acted  as  a  consistent  indicant  ot  achievement  was  the  linear  term.  It 
is,  therefore,  probable  that  the  metric  underlying  individual  testee  response 
had  not  changed  in  any  increasingly  nonlinear  manner,  as  would  be  expected  it 
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the  time  between  testings  systematically  affected  the  metric  along  which 
achievement  was  being  measured,  since  no  such  systematic  trend  was  noted. 

Conclusions 

The  consistent  findings  of  the  analyses  of  the  effect  of  time  of  after- 
peak-of-instruction  testing  on  the  measurement  of  achievement  are  as  follows: 

1.  A  single  linear  regression,  using  only  prior  performance  as  a  pre¬ 
dictor,  was  as  efficient  for  the  prediction  of  later  performance 
as  were  four  regression  lines  that  took  into  account  the  time 
elapsed  between  peak  of  instruction  and  later  testing. 

2.  The  linear  prediction  trend  was  consistently  significant  in  each 
week  of  testing  following  the  peak  of  instruction. 

3.  No  nonlinear  prediction  trend  was  consistently  significant  across 
all  weeks  of  testing. 

4.  No  significant  increase  in  the  significance  of  nonlinear  prediction 
trends  was  observed  witli  the  increase  of  time  elapsed  between  peak 
of  instruction  and  later  testing. 

These  findings  lead  to  the  conclusion  that  there  was  no  evidence  to  sup¬ 
port  the  hypothesis  that  the  achievement  variable  changed  as  a  function  of  the 
time  lapse  between  peak  of  instruction  and  measurement  of  an  individual's 
achievement  level.  The  unidimensional  ICC-based  variable  that  had  been  used  to 
measure  achievement  of  individuals  at  the  peak  of  instruction  seemed  to  ade¬ 
quately  describe  the  achievement  of  individuals  as  much  as  a  month  after  the 
peak  of  instruction. 


IMPLICATIONS  AW  LI 


AT  TOW 


Inrl :  cations 


The  results  of  these  studies  have  implications  for  the  measurement  of 
achievement  using  both  the  pretest-test  and  test-posttest  paradigms.  The  data 
suggest  that  there  may  be  metric  problems  in  the  application  of  ICC  item  para- 
maters  based  on  peak-of-instruction  data  to  pretest  data.  Both  the  item  dif¬ 
ficulty  and  discrimination  parameters  estimated  at  the  pretest  (prior  to  in¬ 
struction)  differed  substantially  from  those  estimated  at  the  peak  of  instruc¬ 
tion.  This  result  was  reinforced  by  factor  analyses  of  item  sets  obtained 
prior  to  instruction  and  at  the  peak  of  instruction;  the  variance  accounted 
for  by  the  first  factor  was  considerably  less  at  the  pretest  than  it  was  at 
the  peak  of  instruction.  The  implication  of  these  results  is  that  ICC-based 
pretest  and  peak-of-instruction  achievement  measurements  may  not  be  on  the  same 
dimension.  Thus,  the  achievement  variable  measured  at  the  pretest  may  be  a 
different  variable  than  that  measured  at  the  peak  of  instruction. 

The  importance  of  this  finding,  if  it  can  be  replicated  in  other  data 
sets,  is  to  call  into  question  the  utility  of  the  pretest-test  model  for  mea¬ 
suring  gains  in  achievement.  If  the  pretest  achievement  variable  is,  in  fact, 
a  different  variable  from  that  measured  at  the  conclusion  of  instruction,  it 
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is  inappropriate  to  compute  individual  or  group  gain  or  change  scores  as  indi¬ 
cants  of  growth  in  achievement.  Such  change  scores  would  be  completely  useless 
in  providing  reliable  estimates  of  growth  in  achievement  levels  because  of  the 
differences  in  the  variables  involved  at  the  two  points  in  time. 

Contrary  to  the  negative  implications  of  these  data  for  measuring  achieve¬ 
ment  in  a  pretest-test  paradigm,  results  of  the  second  studv  support  the  use  of 
an  ICC-based  test-post  Lest  paradigm  for  the  measurement  of  retention,  at  least 
within  the  1-month  time  interval  studied.  Data  from  the  second  .udv  showed 
that  ICC-based  achievement  level  estimates  taken  as  much  as  a  mouth  after  the 
peak  of  instruction  were  consistently  linearly  related  to  achievement  level  es¬ 
timates  taken  at  the  peak  of  instruction.  Thus,  the  data  indicate  that  these 
posttest  measurements  were  on  the  same  ICC  metric  as  the  peak-of- instruction 
achievement  level  estimates.  The  data  did  show  a  level  difference  in  the  achieve¬ 
ment  estimates,  but  this  might  have  resulted  from  design  aspects  of  the  study. 
Should  future  studies  replicate  this  result  (with  or  without  the  level  differ¬ 
ence)  ,  the  data  imply  that  gain  (or  loss)  scores  measuring  retention  after  the 
peak  of  instruction  may  be  meaningfully  determined  using  ICC-based  approaches. 

The  positive  findings  from  the  af ter-peak-of-instruction  data  also  are  in 
support  of  the  potential  of  computerized  adaptive  testing  for  applications  in 
the  measurement  of  achievement.  The  data  indicated  linear  relationships  among 
ICC-based  achievement  level  estimates  obtained  up  to  4  weeks  after  instruction. 
Thus,  even  with  limited  availability  of  testing  terminals,  which  might  be  char¬ 
acteristic  of  adaptive  testing  in  certain  instructional  environments,  it  may 
be  possible  to  obtain  equivalent  achievement  estimates  for  students  tested  as 
long  as  a  few  weeks  after  the  material  was  covered  in  a  course.  This  should 
minimize  the  cost  o:  an  adaptive  testing  system  and  make  its  use  economically 
feasible  for  classrooms  of  all  sizes.  Further  research  will  be  necessary,  of 
course,  to  determine  whether  the  observed  mean  differences  in  student  perfor¬ 
mance  after  the  peak  of  instruction  were  due  to  the  motivational  factors  char¬ 
acteristic  of  voluntary  participation. 


Both  c f  the  studies  reported  above  were  done  within  the  context  of  a  sin¬ 
gle  undergraduate  survey  biology  course.  This  limits  the  generalizabtl ity 
of  the  studies  in  several  ways.  For  example,  the  weak  factor  structure  noted 
in  students'  responses  prior  to  instruction  may  be  due  to  the  fact  that  this 
was  an  introductory  course.  It  is  very  possible  that  a  more  advanced  course 
might  show  a  strong  pr ior-to-peak-of-instruction  factor  structure  which  is 
similar  to  the  peak-of -instruction  factor  structure.  In  addition,  different 
items  drawn  from  the  same  content  pool  were  used  in  the  pr ior— to-instruct ion 
and  peak-of - ins t rue t ion  factor  comparisons.  Future  studies  should  compare  the 
factor  structures  ot  the  same  items  prior  to  instruction  and  at  the  peak  of 
instruct  ion. 

Further  limitations  in  the  constancy  of  the  testing  procedures  might  have 
added  some  biases  to  the  conclusions  drawn.  The  pretest  measure  was  required 
of  all  students  attending  the  first  lecture  of  the  class,  but  it  did  not  affect 
students'  grades  in  the  course.  The  tirst  midquarter  examination  (peak-of-in- 
instruction  test)  was  required  ot  all  students  in  the  course  and  did  have  a 
bearing  on  the  students'  grade  in  the  course.  The  after-peak-of-lnstruct ion 
measure  was  a  voluntary  test  which  allowed  students  to  add  extra  credit  points 
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to  their  course  grade.  In  addition,  this  test  was  a  computer-administered 
stradaptlve  test,  whereas  the  first  two  Lests  were  administered  in  convention 
al  paper-and-penc l 1  format.  These  differences,  hot h  methodological  and  moti¬ 
vational,  may  have  added  some  unknown  amount  of  bias  to  the  results  oi  the 
studies.  Thus,  replication  of  these  studies  In  other  instructional  environ¬ 
ments,  and  witli  revisions  in  the  research  design,  is  appropriate. 
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Table  A 
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Study  v'(  leak /Peak  ravametev  Corresi'vMulewe e 


F  li¬ 

St 

Ailm  t  n  t  s  t  rst 

i  v'u 

St'oiMid 

Admin  i  st  rst  iv'ii 

I  t  om 

' 

P.-u  arte*  ('i 

I’SI  SllH't  <’l 

Numl'pt 

ves  t  * 

1 * 

Vest  * 

7, 

* 

1002 

KI¬ 

.  S2 

1  1 

.14 

sr 

.87 

.17 

7  7 

10  34 

WI 

1 

.01 

17 

.  28 

81 

.85 

.  70 

1  1 

10  IS 

Wl 

1 

.  5S 

- 

56 

.  28 

81 

1.20  -l 

.06 

16 

3201 

W1 

1 

.07 

-1 

14 

.23 

81 

.85  -1 

.  74 

18 

3206 

Wl 

.  7  4 

\ 

51 

.21 

81 

.75  1 

.57 

17 

1216 

Wl 

1 

.27 

- 

67 

.  18 

81 

1.17 

.  60 

1  5 

12  IS 

Wl 

.  S2 

58 

.  1  2 

81 

.80 

.  14 

1  4 

32  17 

KI¬ 

1 

.  54 

- 

17 

.  18 

SI- 

1  . 58 

.  1  1 

4  1 

1241 

WI 

1 

.  1  2 

3 

t. 

48 

.  24 

81 

.01  2 

.00 

1  7 

34 1 4 

Wl 

.SS 

> 

20 

.  32 

81 

1.40  1 

.06 

10 

1651 

W2 

.81 

2 

'3  1 

.  4  4 

82 

.05  7 

.  U 

5  7 

IS  1 2 

W2 

.  74 

- 

66 

.  1 1 

82 

.87 

.  (>  < 

1  1 

1000 

w; 

1 

.  > . 

77 

.  38 

82 

.oo  1 

.17 

16 

4006 

KI¬ 

.84 

- 

50 

.  1  (> 

SI 

1  .05 

.10 

3  1 

r  i 

40  36 

WI- 

1 

.24 

- 

61 

.23 

81 

.08  -1 

.  30 

1  7 

4044 

WI¬ 

.80 

- 

1  2 

.38 

81" 

.80 

.  60 

1  l 

4  220 

KI 

1 

.  16 

- 

4  5 

.  38 

81 

1  .64 

.  07 

1  7 

42  IS 

WF 

.8  1 

1 

54 

i 

.  r. 

SI 

.81  1 

.  4  7 

<  1 

*W~W  1  nt  or 

(Just 

t  ('! 

10 

/(>  ;  j 

'-S|ir  1  nn 

Quart  pi  10 

76;  1  - 1 '  t 

1  st 

MUlqu.ut 

e  l  I"  x 

Sill  1  ll.-lt 

i  on ; 

7-Seeon»l 

1  MUI^iisite 

v  Ixsmin 

st  t ('ii ; 

I'-  Fin  a  1 

Ix.-itii  1 11,-lt 

t  on 

t 


«!(<- 


OJ(^HHHO(NinfsOCCvTH(Nr4vC'^v1  »Ol  | 
(NOr-(H(NHHHHH(NHCH(NCJHC<NH  I 


vT  fM  in  vt  on 

*— 4  r— 4  r— 4  r— 4  r— 4  <7  *— 4 

I 


<f(*>rvvtN0>(N<Sr4inc'0'<N(^  I  r\j 
0*-4Or-4.-4Or~lO00«— '*-<*-*0  l  O 

I  I  I 


rMinC'0CC'0'^CH0CC<NC<N»\MoC>^  I  <Nr~J 

I  I 


vPvT(N(N(M(NiC'-<(T»infv<SC(N(N(r\r  I  O'  OC  ^ 
(\r-^HHr-l(NOHHr-<<N(N(NHn|(N*-t  I  O  <7  »-4 

00  CC  (N  O'  C  C  ^  O'  ^  ^  \0  <"  O'  I  CC  <N  vC  vT 

*  *— l  r-4  *-<  <N  O  O  l  0  0  0  0 

I 

i^-  h  vt  m  O'  oc  oc  si  on  oo  »  o  vr  vc  i  on  o  *-4  •'n 

(NN(r'(N(N(NHfN(N(v'(^rsJp-i^(*''  I  t- i  *-4  *-4  O  O 

I  I  I 

■4  (M  O'  vt  OC  in  c  h  in  cc  <t  m  I  on  c»n  r-  r~* 

(n(\(N^(v'(NH(N(N('J(n^(M(S  |  o  o  *-*  O  O  —• 

I 

C  \T  in  c  vC  h  n'  OC  lf>  ''A  ^  vf  *  O'  m  m  r-.  \c  on 
vtOjHNlNlNHHO^l^^^  I  O  O  O  O  O  —•  O 

I  I 

CO  in  IT  (M  H  O'  «n  vj  fv  <?N  ^  \  H  ^  ^  u'  r.  O  ^ 

hhCC<NhhhiN(N-h  |  r-^HC(NO'-lH 

I 


O'*  <Jv  <t  m  h  O'  in  <n  O' 


(N  O'  m  it  \  <t  o  o>  v.t  <m  n  o  O 

tN  ^  r- <04  [  *-*00»-«Ot-'<0'-<0 

I 


O  vC  <N  <t  O  on  O  vT  o  vx'  l  VC  <S  m  M  00  n-i  \T  m  lA  c 
mnniHr'iNr^fN^ni  \  .-4<nOomOOOOO»-« 

OOOCT»r^r-<Onsjr^  l  O  ~t  ^  h'  (N  (N  r.  r>  n’  rs 

(Nnj-iH^rviCnini  lOOOOO— ‘OOOOO 

I  I  I  I 

o  O  on  00  vr  oc  O  o  I  00  OC  M  ’  *  *o>  <"M  vj  O  *-<  0>  O  O 
NHNC(^<N(N<*1  I  000  0000*~»0000 

III  III 

M?  (N  vO  vi'  vj  > — *  r— 4  \  vJH(v'^s}(Nr'(N\t  st  1-1  C  ^ 

O  > — 4  • — 4  O  on  Ovj  r— <  (  CC<NhNO<NhCmC^C 

00  iC  M  <n  in  cp  I  N  (S  vt  00  00  c  rH  in  M  O'  00  •  -  VI' 

•—<  *~-4  c\J  *-4  csj  <  I  r-JOOOOosJO— *— 'O^^O*-- 

I 

on  if  sj  m  \  c  in  r-i  in  ,'n  c  C  ■'l  C  cV  nj  vr 

O  »— 1  (N  h  in  I  .-4  ,-4  o  O  <7  <7  »— *000*-<000v“' 

I  I 

00  on  00  Kf\  \  in  C  in  (M  00  M  <t  <N  O  ■'t  O  O  fN  O'  N 

(N  •-<  (N  r-^  I  O  O  O  — <  •“ 4r-^0— *0'“*0— ‘OOOO 

7  i 

h  <t  i  inrxCtSsjm'rNrNin  o  oo  <t  <t  -i  o 
IN  H  H  I  o  O  «-4*--<OOOt-4»'n£nr-*cnO<NOC7C'‘ 

I 

04  ir\  |  dHiA(vii3NiOnH(v'inO(v,Of|«oc^'i(N 
O  oj  l  r-^OOOOQOOO^-HO^OOOOO 
I 


on  I  M  (N  c  ^  'O  IN  H  <N  H  O  lT  (N  C  C  ^  fv  A' 

OM  I  ooooooooooooooooooo 

III  I  I 


I  HH^-4HMNCNO>^rN^vtVl'^C'(v'(nOi 

i  ^^o»-iooooo^^»-*o<noo<noo^ 

I  I  I 

r-<  m  <n  vt  invCNflOOvC-^M  on  v.t  in  vT  00  O  O  *-« 

r— 4  M  *~4  •— 4  f-<  •— 4  •— 4  r— 4  M  OVJ  (>J 


Pretext  and  .Vld'V.wrter . 


N.*v  v 


I 


prsrv; 


1  Pr  .  Fd  Aiken 

Navy  far  normal  RAP  Can  tar 
San  Piag««t  v.' A  d.W 

t  pr .  Jack  R.  Bor at mg 

Provost  \  Ac  ad  era 10  IV.m 
U .  .  Naval  l\>at  graduate  School 
Mon ter  ov  ,  v  A  d  O'40 

1  IV.  Robot* t  IVtMli* 

Cod  »'  N  -  ’  t 

\’AV VRAEQUIPCFN 

Or  l  and© ,  Ft  > 

t  MR.  MAUN  1C t  C  Al l A KAN 
Para  «'J*i 

pwre.-.o  of  Naval  tYrsonne, 

Washington,  IV  v\> 

1  pr  .  Richard  Ftster 

iwrtment  ot  Administrative  Sciences 
V « v  a  l  l\' 5 1  ad  ua t  e  'V  hoo  l 
•V'1  »  *  ov  ,  ,  A  d  W«0 

i  or.  pa  i  ffpfrky 

'  \\  \  PI  h.VWI  l  RV>  CFNTFR 

II  >  1  NCR) ,  C  A  KM  S3 

y  Pr  .  PrtUl  Foley 

\  v  y  !'»'•  sonnet  RAP  Center 
'i\  Plage,  CA  ''-MS' 

1  M  .  John  Ford 

Navy  Personnel  RAP  Center 
S.«n  Pi  ago,  CA  dp  IV 

1  v  APT.  P.V.  GRAGvl ,  VC.  P  A 

Ml  AP,  .*  C  ION  ON  'U  PIC  Al  I  l\  C  AVION 
UN'tFOV  '  ■>:  RV  10FS  UN l V .  ,Y  T1  f 
Mi;  V  IK  NCI  l 

foil  •  ARLINGTON  ROAP 
I'THF:  'A,  VP  ,\V1A 

t  Cl'S  Robert  .  Kennedy 

Naval  Aerospace  fYdU'dl  and 
Rear  arch  tab 
k'«  'd4'; 

Mew  Or  le.ma ,  l  A  ?OYjU> 

t  Pr .  Nortr  a  J,  Kerr 

Chief  of  Naval  Technical  Training 
Nav»;  Air  Station  Men  phi  a  \ 's> 
Millington,  IN  'SPS4 

1  Pr  .  Leonard  Mocker 

Yiv  >  Personnel  R.’vO  Cent  er 
an  Piego,  PA  OP  IV 

t  CHAIRMAN,  l.F APFRSHIP  A  l.AW  PFPT. 

PIV.  vV  PR.iFtSSTPJlAt  Pl: VF LOfttMfkT 
U.P.  NAVA l  ACAPFMYY 
ANNA  KYI  IS ,  Ml>  PY40P 

1  Pr .  William  L.  Mi  toy 

Principal  Civilian  Advisor  for 
Fd  no  lit  ton  ml  Training 
Naval  Training  Command,  Code  PPA 
Pensacola,  FI  «.'MOS 

\  cAPT  Richard  t.  Martin 

USS  Francis  Marion  (l.rA~;4dY 
F IN'  New  York,  NY  0<JVM 


Pr  .  Janes  MeiVlde  t 

Code  $01 

Navy  Per sonnet  SAP  Center 
.'sin  Piego,  CA  OPYV 

Pr  ,  Janes  McGrath  1 

Navy  Personnel  RAP  Center 
Code  tOb 

Ivin  Pw  go ,  CA  dPtV 

1  PR.  Will  l  A'*  VNt  AGUE  \ 

LN'IV 

UN  l  \T  RS  1 1  Y  ''I  PlTTSbURGM 
JO  O'HARA  M  KltT 
PITTSBURGH,  PA  IV1  l 

1  Command  ing  Officer 
Naval  Health  Research 

Cent er 

Attn i  library 

San  Piego,  C\  OP  tv 

1 

1  Na\  al  Medical  R\  '  Civmmand 
Code  4*i 

National  Naval  Medical  venter 
IV  t  head  a,  “'  pool* 

‘  ibrary 

Na\ \  IVr  sonnet  RtP  Center 
'sin  Pi  ego ,  ca  o  ' !  V 

h  Command  ing  Officer  . 

Naval  Research  Laboratory 
Code  PbP ' 

Washington ,  IN  PO'dO 

1  PI  F  1C h  OF  v  ti  ll  IAN  f;  RSONNl  l 
VvYPI  0V 
PFPT.  OF  vm  NAVY 
WASHINGTON ,  V  .V  top 

1  JOHN  CI  St  N 

CIUFF  NAVA  l  1  PAY  ATI  ON  1  ! 

VRA'Nl \v  .SUPPORT 
PFNSArVU,  M  L'spc 

1  Payohologiat  I 

.'NS  branch  Office 
•< os  Srjnner  Street 
lV>ston,  A  OOP  IP 

1  Psy civ  legist 

ON R  branch  Office 

f' M  S,  Clar  k  Street  I 

vTitcago,  tl  bOSOM 

I  Office  of  Naval  Research 

Code  POP 

Arlington,  VA  PPP1T  1 

I  Code  4  lb 

\'f  t  ice  v't  Nav  .  1  Resear  ch 
Ar  l  Ing  ton  ,  V  A  ,\V  ' 

1  Office  of  Naval  Sesaarct' 

Code  4 l * 

?*'•'  N  .  -Nil  no y  S'-t  r  eet 
Arlington,  \v  ,'PPV 

'*  Per  Si'nnel  v  P  aining  Research 

Vi'oae  kbgl 

Office  of  Naval  Research 
Arlington,  VA  PPPM 


Psyc k>l vvg  ist 

OFMCF  'N  NAVAl  RIS1  ARCH  bRANx  • 

PP1  vUP  NARYl  HL'Nt  Rv'A !' 

Vv'MVN,  NVi ,  1*»TM  INC.l  ANP 

Psy ,'lc  leg  i  st 
ONR  branch  \M‘1  ice 
tp  tp  last  C.r  »  en  Street 
Pasadena,  CA  01101 

Scientific  Pi  rector 

'iff ice  ot  Naval  Research 

Scientific  liaison  Group  Towyo 

Ann  i c an  FYwbassv 

AIN'  San  H. incisco,  »A  ot  ; 

Office  ''t  t  Me  Chid  Naval  .Y*’»  at  ions 
Research,  IVv  elc|vsent  ,  and  M«kilrs  !'».*• 
vor~nv> 

Waahtngtv'n ,  J\  ,Y  tso 

Scientific  AdviavU  to  the  O-ici  ot 
Naval  Per  sonne  l  v  P»  •  ••  P  ' 

Naval  INir  caw  ot  IVr  sonne*, 

S\',nm  4  4  10,  Arlington  An  nr a 
Washington,  ,\  ,\'t‘0 

IT  Frank  c.  IV tin',  use,  t  \K  vpm.p' 

Code  V't 

Naval  Aerospace  Mpdieal  kesea*el*  labors! 
Pensacola.  II  L'M'S 

PR.  RU'HARP  A  IN'l  ;  IN 
ACAITM1C  ,  ,VM'  •:  ;\,  Cl  M  s 
U.S,  NAVAl  A,  AOLMl 
AWAIN'V  IS  ,  ,'ts." 

Roger  **  .  Remington ,  Pt  p 
v'ode  l*'0 

NA‘*Rl 

Pensacola,  II  '.P'.ps 

Pv  .  IVr  var  d  R\s*\  and 

Na \  v  Per  scinrel  RAP  Cent'  * 

'Van  0l  ego,  ,  A  opt*'* 

Mr.  Arnold  Ruben  stein 

Naval  Personnel  Sunvr  t  c .  ’  -o o  v  v 

Naval  Fat  cv  i  si  0'Vs*»an,i  vO'i*.  0*4' 

Rocs  to 4 4,  sYvstal  P \  a  ; a  f  • 

PPPt  Jefferson  IVv  is  •■ig?«.  >v 
A>  l  ington  ,  \  A  .V  5b0 

Or.  WN'rth  Ivanland 

vTvief  ,'t  Naval  FC.k'at  i«"  '>  .n  -  v 

Oode  N  *v 

NAS,  Pensacola,  Ft 

A  .  A .  SJOHOt  m 
TFCM.  SUPP'Rl ,  00 Pf  A'l 
NAVY  Tl  RSPNNM  R.\  P  C1NVIR 
SAN  PUvY,  *  A  O  ' ISO 

Mr.  Robert  :v*it»r 

'ft  tee  cl  '>*tet  of  Naval  .Yc  at  ic*'S 
,'r-05*i 

Washington,  ,\  ,V  tsp 


Prog*  am s 

i  Pt  .  AVfrtvl  F.  iNn*vc 

Training  Anatvsia  V  |\#l«>itton  x  - owp 

t  rite.' 

Pept  ,  ot  t  Me  Nav  v 
.Nlarvlo,  FI  t,\<  I  t 


> 


i 


1 


1 


1 


1 


1 


1 


1 


1 


I 


1 


I 


1 


1 


1 


Or.  Richard  IVrenaen  y 

Navy  Personnel  RAO  Center 
:’>an  Diego,  CA  V1V 

COR  Charles  J.  Theisen,  JR.  NSC,  USN 
Head  Human  Factors  Engineering  Oiv.  1 
Naval  Air  Development  Center 
Warminster,  PA  tS97*» 

d.  Gary  Thomson 

Naval  iVean  Systems  Center  1 

Code  7U<? 

Can  Diego,  CA 

Or.  Ronald  We  it  {man 

Department  of  Administrative  Sciences 
l).  S.  Naval  Postgraduate  School 
kvnterey,  CA  dtdUO 

OR.  MARTIN  F.  WtSKOFE 

NAVY  PERSONNEL  RK  0  CENTER 

SAN  DIEGO,  CA  <tfiv  . 


1 

Army 


Technical  Director 

U.  Army  Research  Institute  for  the  1 
behavioral  and  Social  Sciences 
•'I'M  Eisenhower  Avenue 
Alexandria,  VA  ??  *  > 3 

MO  UcAREUE  X  ?th  Army 

O PC HOPS 

USMREUF  Director  of  GFD 
API'  New  York  0040  t 

1 

Lv  Cl.  Gary  rioedorn 

Training  Effectiveness  Analysis  Division 
Army  TRAOiV  Systems  Analysis  Activity 
White  Sands  Missile  Range,  NN  SS<W  y 

PR.  RALPH  DUSEK 

U.S.  ARMY  RESEARCH  INSTT TOTE 

M’-'l  EtSENHOWER  AVFNUE  y 

A l  E  XANOR  IA  ,  VA  ,Y  m 

Mr,  Vyron  El  Sc hi 

U.S.  Ar  iy  Research  Institute  for  the  ^ 
Social  and  behavioral  Sciences 
l'iVM  Eisenhower  Avenue 
Alexandria,  VA  ?pttt 

t 

’V  .  Ed  Johnson 

Army  Research  Institute 

•' ' '  1  Ei  senhower  Mvd . 

Alexandria,  VA  ,VH: 

Dr .  Michael  Kaplan  y 

.  .  ARMY  RESEARCH  INSTITUTE 
•"!  EISENHOWER  AVENUE 
ALEVA  NOR  I A ,  VA  ,V  M \ 

Or.  Milton  S,  Katl  ^ 

Individual  Training  A  Skill 

Evaluation  Technical  Area 

U.S,  Army  Research  Institute 

S0(>1  Eisenhower  Avenue  1 

Alexandria,  VA  72  U t 

Or.  Beatrice  J.  Farr 

Army  Research  Institute  (PERI -OK )  y 

M'Ot  Eisenhower  Avenue 
A]  »*x  undr  I  a  ,  VA  ...  tt* 

t 


Dr.  Hilt  Haler 

U.S.  ARN\  RESEARCH  INST l TiTT t 
*001  EISFNHv'WER  AVENOF 
ALEXANDRIA,  VA  22**\ 

IV.  Harold  F.  O’Neil,  Jr . 

ATTN:  rFR l -OK 

*00 1  F 1SKNMOWFR  AVENUE 

ALEXANDRIA,  VA  77  ti t 

1 

IV.  Robert  Rosa 

U.S.  Army  Research  Institute  for  the 
Social  and  behavioral  Sciences 
SOOt  Elsenhower  Avenue 
Alexandria,  VA  t 

tv.  Robert  Ha  sm  or 

It.  S.  Army  Research  Institute  for  the 
behavioral  and  Social  Sciences 
sOOt  Elsenhower  Avenue  1 

Alexandria,  VA  rVUj 

Director,  Training  Development 
U.S.  Army  Administration  Center 
ATTN:  Or.  Sherrill 
Ft.  benjamin  Harrison,  IN  4&MS 

Dr.  Frederick  Steinheiser 
U.  S.  Army  Reserch  Institute 
soot  Eisenhower  Avenue  , 

Alexandria,  VA  < '  J 

Or  .  Joseph  Ward 

U.S.  Army  Research  Institute 

soot  Eisenhower  Avenue 

A  *  ex  and r  la  ,  VA  ,V  t  t \ 


Air  Force 


Air  Force  Human  Resources  lab 
AEHRl  PFD 

brooks  AFb,  TX  7SPis 

Air  University  Library 

Max w^l  1  AFb,  Al  :mi; 

Or.  Philip  IV  l  eo 
AEHRL  TT 

Lowry  AFb,  00  SOD  to 

OR.  G.  A.  FCKSTRANO 
AEHRl  AS 

WRIGHT -PATTERSON  AFb,  vNi  »snu 

Or.  vVnevieve  Haddad 
rrogram  Manager 
life  Sciences  Directorate 
AE  (VSR 

IVllllng  AFb,  IV  A'  5  50 

CnH.  MERCER 
CNF T  l  1  A  ISON  OFFICER 
AEHRl  FI  Y  INC.  TRA  INING  OIV. 
WILLIAMS  AFb,  A?  SS2r 4 

Or.  Rosa  1.  Morgan  l  AEHRl  ASR' 
Wright  -rail arson  AFb 
Ohio  4*4  tt 

Or.  Roger  Dannell 
AEHRl  TT 

Lowry  AE  b,  CO  d  V  10 

Personnel  Analysis  Division 
Hv,'  US  AT  orx\A 
Washington ,  IX  A' '  to 

Research  IVanch 
AFMIY  DPMYP 

Randolph  Arp,  n  , ms 


Or.  Malcolm  Rer 
AEHRl  rEO 

brooks  AFb,  TX  TRJt* 


Mar  ines 


H.  William  Greenup 
Education  Advisor  iFDSt) 

Education  venter  ,  MCDEC 
Ouantleo,  VA  .\'t  :<i 

Director,  Office  of  Nanix'wer  Utilisation 
HO,  Marine  Clirps 

IVb,  bldg.  A  VO 

Ouantioo,  VA  ,Vt*4 

OR.  A.L,  SI  AT  Kv'SkY 
SCIENTIFIC  ADVISOR  uVl'E  RD- 1  t 
MC,  U.S.  MARINE  CORPS 
WASHINGTON,  A  A'  tSO 


Co  a  st  Guar  »1 


Mr.  Richard  lanteman 
PSYCk'l  vV.Iv  At  KFSFARv  H  v«  -P  -1 
U.S.  COAST  GUARD  HC 
WASHINGTON,  IX  ,V*dO 

Or  .  Thomas  Warn 
U.  S.  Coast  Guard  Institute 
P.  0,  Substation  tH 
Ok  l  ahoma  City,  OK  * ;  t 


Other  IV 0 


Defense  IVciwient  at  ion  Center 
Cameron  Station,  bldg.  * 

Alexandria,  VA  ,V;t4 
At  t  n :  TV 

Or.  Dexter  Fletcher 

APVANCt  0  RE’d  ARCH  PROJFCIS  Avl  h  A 

1400  WILSON  bLVT. 

ARLINGTON,  VA  .Y  V  i 

Dr.  William  Graham 
Testing  Directorate 
MEIYUN 

Ft  .  Sheridan  ,  H  so  '  i  * 

Military  Assisi  »nt  for  I>  aining  and 
Personnel  TWohnolog\ 

Office  of  the  I4ider  'Secret  ary  of  IVfcnse 
for  Research  A  FYiglneeryng 
Ro.vs  10  t/O,  lire  TX*nt  agon 
Washington,  ix  A' tot 

MAJOR  Wayne  'V liman,  unAf 
Office  of  the  Assistant  Vet  el  ar  \ 

Of  IV  t  ense  v  MR  AM  ' 
tbOtO  The  Pent  agor 
Washington,  !X  A'tOt 


r 


I 


Civ il  Gov t  I 


Pr .  Susan  CM  pm  an 
hssic  Skills  Program 

Hattons!  Institute  of  Education  » 

1200  19  th  St r re l  N W 

Washington,  DC  20206 

Dr.  William  ^orhan,  Dir actor 

Per  sonnel  RAD  Can  tar 

Office  of  fVraonnel  Managment 

1900  E  Street  MM  y 

Washington,  DC  20R19 

IV.  Joseph  l.  Llpson 

?ivisio:»  of  Science  Education  1 

Room  k-b  1g 

National  Science  Found at  ion 
Washington.  DC  20S90 

Dr.  John  Kays  I 

National  Institute  of  Education 
1200  19  th  St  r cat  4 
washing ion,  DC  CO.V 

Dr  .  Art hur  F>* im*d  y 

National  lnlitute  of  Education 
1?tV  19th  Street  Nw 
Washington,  DC  CO.’."' 

Dr,  And  raw  R.  'K>lnar 

Science  Education  tVv  .  y 

»hd  Research 

Nation  i'.  Planer  Foundation 
..ash in,; ton ,  DC  C0S50 

Dr .  Lalithj  p.  Sanathanan 
Environmental  Impact  Studies  Division 
Argonnc  Sat lonal  laboratory 
•’CD  3.  Cuss  Avrnue 
Argonne ,  II  bC4  19 

Dr.  Jeffrey  Schiller 

'•it  ion  j1  Institute  of  Education 

1C  V  rth  St .  NW 

...i  .v:  mg  ton,  DC  20223 

Dr.  Ihomas  G.  Sticht  1 

D-~»ic  .  »ills  Program 
‘-ttlonal  Institute  of  EJucat  ion 
12DD  19th  Street  NVi 


..  js.vingtori , 

PC  20203 

Dr.  Vera  W, 

tVr  y 

Personnel  HAD  Center 
Dffice  of  Personnel  fliriag  ient 
190  C  f  Street  NW 
Washington,  DC  amis 

Dr.  Joseph  L.  Young,  Director 

*tefnory  t  Cognitive  Processes  1 

N  ition.il  Science  Foundation 

washing ton ,  PC  conso 


Von  Covt 


Dr.  F ,r  1  A.  Allwlsl 

‘w.  AFEiRL  iAFSO  1 

crooks  AFP ,  TX  /3c  IN 

v •  ft* ling  P.  Anderson 
(rilversity  of  CopcrD.agen 
Stud  lest  rued  t 
CopenhM&en 

DENMARK 


•  psychological  research  unit 

Dept,  of  INrfense  i  Army  Office' 

Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

Dr.  Alan  Haddeley 
Modi  al  Mesearch  Council 

Applied  Psychology  UR  It 
19  Chaucer  Road 
Cambridge  CB2  2EF 
ENGLAND 

Dr.  Isaac  He  Jar  1 

Fduoal lonal  Testing  Service 
Princeton,  NJ  OHM SO 

Dr.  Warner  Hi  rice 
St  reitkrae fteaml 
Ko  sen  berg  s  100 
Bonn,  West  Germany  P-S  ftV 

Dr  .  R.  IWrel  Ek>ck 

Department  of  Ed uc at  ion 

University  of  Chicago 

Chicago,  II  606 IT  1 

Dr.  Nicholas  A.  Bond 

Dept,  of  Psychology 

:Wi  or  amen  to  State  College 

600  Jay  Street  1 

Sacramento,  CA  9S gto 

Dr.  David  G.  lowers 
Institute  for  Social  Research 
University  of  Michigan 
Ann  Arbor,  Ml  MbIOb 

Dr.  fN>6ert  (Yemtan 

Riser  lean  College  Testing  Programs  , 

P.  0.  Ho*  IM 

Iowa  City,  I A  VCV 

DR.  C.  VIC  DM  Hl»N DfRSv'N 
WICAT  INC. 

UNIVERSITY  PLA.’A.  SUITE'  to 
M bP  SO.  STATF  ST. 

'RFM,  IT  gao«. 

Dr.  John  B.  Carroll 

Psychometric  Lab  1 

Uhlv.  of  No.  Carolina 

Dav  ic  Pull  >1  1A 

Chapel  full,  'ft  .'79 14 

Charles  tyers  Library 

Livingstone  House  l 

Livingstone  Road 

Stratford 

London  FIS  CLJ 

ENGLAND 

IV.  John  Oil  or  im 
Lit ton- del  Ionics 
tV>*  1CB6 

Springfield,  VA  021  SI 

IV.  Kenneth  t.  Clark  y 

Co  liege  of  Art  S  .1  Sc  i end e s 
University  of  Rochester 
Hiver  Campus  Station 
Rochester,  NY  1 4bC  * 

Dr .  Norman  Cl l ff 
IV pt  .  of  Psychology 
iHiiv.  of  :o .  California 
University  Park 

Los  Angeles,  CA  90007  , 

Dr  ,  W 1 1 1 1  .tm  Co  ffm  an 
Iowa  Testing  Programs 
imiverMty  of  Iowa 
Iowa  City,  1A  s.'CRC 


Dr .  All  an  M.  Collins 

fck>ll  ttvranek  i  Netanan,  Inc. 

90  Moulton  Street 
Cambridge,  Ma  02 ms 

Dr.  Meredith  Crawford 

Ur  pertinent  of  Engineering  Administration 
Gaorgv  -ashing ton  University 
Suite  30S 

2101  L  Street  N.  W. 

Washington,  DC  2003? 

Dr  .  Hans  Cronbag 

Ed uc at  ion  Research  Center 

Univer  sit  y  of  Leyder 

Boerhaavelaan  C 

Leyden 

rhe  NETHERLANDS 
MAJOR  I.  N.  EVt'N IC 

cawdun  reacts  n *s.  sm.il  • .. 

UP’  AVKNUF  USA'.' 

TCMOtilV,  ,'NtAHlP,  i  WA.'A 

Dr.  Leonard  Eclat 
1 1  ndgui  st  Center  for  ^Vusurteei  X 
University  of  low.. 

Iowa  City.  1A  S2CRC 

Dr .  Richard  L.  Ferguson 

T1\e  American  College  Test  u  .  is. 

P.C.  fk.*w  16g 

Iowa  City,  I A  *CC<*C 

Dr.  Victor  Fields 
IV pt  .  of  Psyctology 
M'ntgumer v  College 
Rockville,  "P 

Dr.  Gerhardt  Fischer 

llebigasse  • 

Vienna  1012 

Austria 

Dr.  tVnald  Fitzgerald 
IRllversity  of  Vw  ftr.Uio 
Arm  id  ale.  Sew  South  Wales  »•  ) 

AUSTRALIA 


Dr  .  Edwin  A.  Fie  is*  -,.r 

Ad  vane  eel  Rescarcl  •  vn.*  .»  s  s 

Suite  900 

•*<10  Fast  West  Highway 

Washington,  IV  ,Wiu 

Dr.  John  R.  Frederiksen 
»\'lt  tV v. mew  \  \e*e 
90  Moulton  Street 
Cambridge,  MA  C.'l  <S 

DR.  RtlOERT  GLASER 

LRIX' 

UN  W  R^'TY  CE  PI  r  I . 

19 Ad  ChAHA  'THEFT 
PITTSBURGH,  PA  is m t 

Dr .  Ross  Greene 
CTD  McOawi  Mill 
l>*l  ^rtc  Research  Park 
Monterey,  ,A  Otdkp 

Dr.  Alan  Gross 

Center  for  Mvanocd  tiM>  it  Ediwatlon 
City  Univer  sit  v  of  Vw  Yo»  k 
New  York ,  NY  ».'•*  -*6 

Dr.  1k>n  Mambleton 

5o  hoc l  of  Flue at  ion 

diversity  of  »sse,  *  ;setrs 
Amherst,  MA  OIC'C 


1 


IV.  Chester  Karris 
School  of  Education 
Uni varsity  of  California 
Santa  Barbara,  CA  4  1 10b 

Pr .  Lloyd  Humphreys 
IV  part  want  of  Psychology 
University  of  Illinois 
Champaign,  It  M820 

l lbrary 

MomKRP  Western  Pi  vision 
P’Rv*  Berwick  IVive 
arise l  ,  CA  HV1 

IV.  Steven  Hunk* 

IV par t went  of  Education 
University  v't  Alberts 
tvi  onion ,  Alberto 
CANADA 

Pr.  Far  l  Hunt 
tVpt  .  ot  Psy  etiology 
.>*1  varsity  of  Washington 
»t  t  le  ,  WA  c  HO* 

r  .  Huynh  Huynh 
vi  .irt  'UMit  of  Fducat  ion 
University  .'t  .*outh  Carol  Ina 
CoWnbii,  SC  V*’ 

jv  .  Carl  J,  Jen sew a 

.■ail  illdt  t  rol  lege 

Ken  loll  Green 
washingto  "  .  i\  .VO.'.' 

v,  \r-Mol>‘  F.  KanarlcK 
Hoticyw*  It  ,  Inc, 

K  i  ift»  woy  Pkwy 
"i  "Hr  ijv  »  is,  *’N  •»***  t 

V .  John  A.  Keats 
.  ■  ■  .  v  >  ■  ■  i  *  v  ,*  t  ',«•  «v  ast  l  e 
*  *,v  .-  tli,  *.*  *.  %'uth  **  ties 

Alls  THAI  l  A 

Mr  .  i«rl  m  Kroner 
nr  via  Golet* 

,  tea .  C  A 

UVl.  C.R.J.  LAFLEUR 
y  s\  P'.NF  l  Am  It  P  Ki  SF  ARCH 
*1.  \Al  't  F  F  N.*F  HvJS 
10t  .  A.  I  DY  'i’.Yr 

,  I T \« A,  1  ANAPA  K  t A  PSP 

't  .  *ti  .•  t.  u*  l  l  ev  \  le 

v;  irtme-it  'i  r.ti.x  .*t  mnsl  Psychology 
4i«  iv  er  s  1 1  v  x't  Illinois 
V*  IM|V»lr,l»,  11  t  t.V.l 

Faculta  it  Vcule  We  ten  ik*  happen 
Ki  iktuniversit eit  Groningen 
iXhM-  lN»ter  i*>  ;ostreat 
irorungrn 

nf  rm  rlah.ys 

v .  Hebert  linn 
Col  left  r  of  Fit  ik*  at  ion 
m iversity  ot  III inois 
Urbane,  »l  Mfrtt 

:v  .  Frederick  U,  lord 
‘.Iwcet  lo»*  «\  Vesting  .‘<erelce 
Princeton,  NJ  'Ai*p 

Pr .  fcvbert  A.  .Okie 
Mviwan  hr  tors  Research,  Inc, 

A? AO  Cortona  IVive 
»nt  i  Kirbara  Research  Pk  . 


t  Dr.  Carry  Marco 

Educational  Testing  Service 
Princeton,  NJ  08 A SO 

t  Dr.  Scott  Maxwell 

Department  of  Psychology 
University  of  Houston 
Houston,  TX  rro.'H 

t  IV  .  Sam  Ha  y\> 

Loyola  UUverslty  of  Chicago 
Chicago.  IL  bOfvO  t 

t  Dr.  Allen  Hunro 

Un  l  v  .  of  So.  California 
INthavloral  Technology  labs 
AT  1 P  South  Hope  Street 
l.o  s  An  ft  e  l  e  s ,  C  A  OP  00  * 

t  Dr.  Melvin  R.  Novick 
Iowa  Test  ing  Programs 
l »n iversity  of  low,* 

Iowa  City,  l A  s.'.'a,' 

t  Dr.  Jesse  Oriansky 

Institute  for  V  ten  sc  Analysis 
HOP  Army  h.tvy  IV*  \  \ , 

Arlington,  VA  '  *.'P.' 

1  Dr.  James  A.  Paulson 


»V rt  land  M*.  ate  lHiivcrsit 
P.v\  IV  v  ’*>t 

tVrtland.  v'S 

ur.vr 

MR.  LUIGI  PE  TRULLP 

C4v»  N.  FIV.t » 

s\'P  V N t  t  l 

AKl.  l  Nv‘.  1\'N  ,  NA 

***** 

PS’.  8Tfir.v  M. 

I'lkt 

4>t*vs'  'Villas 

Av  cr.ue 

uolden  Valley 

,  KK  •.•■atb 

I'N .  DlAKt  M. 

NA"  t  Y  Ns  *  : 

K-K  RE8FAR  '• 

v  cYC'TEM  s': 

tds  S.APM.A 

r  U: 

MALI DU,  »  A 

'  Pps 

UV.  RET.  M, 

P  ll  A 

KAUv't 

\u  :t  ; '•  \  s  i.*  s;:  :  "... 

IV'UTACM  1«0 
s  t  »K'fJN  1,  *P  KV  AM 

t  Dr.  IV ter  D.  R*.h1 

Vcial  :*c  le  ice  Sesear.t  v'oiaicll 
;*P  Thir\l  Avenue 
New  York,  \Y  100  H 

1  Dr.  *iark  p.  Revkase 

Educational  Psycfvlogy  IV, 't, 
vAiivi  sity  »*f  Rt.i.viir  t  -P'liwMa 
IP  Hill  Hall 
Columbia,  MP  hVOl 

1  Dr.  Fred  Re if 
.  i  :>A'*f 

c  o  Physics  ,Vpa»  t  *cit 
University  »' t  valiforni<< 

•Vrkely,  CA  \V 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
Thomas  Jefferson  t  .  NV 
Wash  l ng t  on  ,  PC  VPP  * 

i  IV.  laonard  l.  Rosenbain,  v^talrmat 
Department  of  Psychology 
'V'»* t ft.wet  y  College 
Ros'kv  l  lie,  MP  \Asp 


l>* ,  tV  n  st  Rothkopf 
IV 1 1  laboratories 
600  Mountain  Avenue 
Murray  Mill,  NJ  07 *) * k 

\  IV,  IV n aid  Rubin 

Fducat  lonel  TVs  ting  8ervi»'e 
Princeton,  NJ  08a HO 

t  IV  .  Larry  Rudner 
vlallaiklet  College 
Kendall  C»reen 
Washington,  IV  .VPP.' 

1  Dr,  J,  Ryan 

Pe  part  went  v't  Fducat  mn 
IV iversity  v't  JVuth  vatolin* 

Co  1  imtMa ,  $C  .'0P08 

t  t'RvR  .  FIVIKP  tAMtJIMA 

DEPT,  v*  P8YC»K'liVY 
.All.  S8 1 TY  PF  TENNF88FI 
W  v.  ,  m 

\  DR.  HOD  CRT  J.  SFIPEl 

N  \  . 

NUN  S'  tv' 

:pp  «A  v|f  l\,  *  ,'N  St . 

Alt  XANDS 'A,  l A  .V  ta 

1  'r  .  Ka.-.i,'  Mi  gee*  a  su 
iV l versify  ct  !\'tu>ku 
.Vpa»  t  i»*nt  ,*t  t  dvk'af  U'nal  F*sy.  fk»l 

K.IMiClM  ,  iMI.Ul 

JAPAN 

)  IV.  Fvl w  !  ■  Stiu  kc> 

IVpart  -'ei’t  v't  Psyc  Avlv'fty 
Flv'J'lda  IccJuiv'U' ;  leal  >lh 

vVlarkfo,  M  i.’S’c 

t  Dr.  Kv'bert  .'kith 

JV|v*rtment  v'f  Cv'mputei  Vience 
Nut  ftei  s  i\  .  •  •>  i  \  \ 

Ni**.  !  i  >ev,i.  y  ,  V  . 

V  .  Rvctuvrvl  Snow 
.Vhool  v*t  Fdik  at  iv'n 
M  mforvf  IRllvet  sit  v 
'•1  anfXsr.f ,  v  A  *.'•*  «. 

Dr.  tk'ber t  St rrnberg 
Drpt .  v't  tsv. 'v'lv'g v 

Y  il  t  Ail  ve*  $it  y 

tk'V  1  1A  ,  Yale  '  X  -•  t  U't 
New  'in  r  ■ ,  i  .  ‘P 

'S.  A l  H  t  v  fVFN 

»vH  1  HEKAKt  K  A  Nt  ».H4\t  'A,  . 

HO  “v\'l  VPN  >*.  K!  FI 

vAMHRKXt,  ha  ,V1« 

1  'S.  PAlKlvK  .0PPI.N 

.  N  .  .  :  *  ■  •  A  •  'A  .  A. 

TNI  -vVlAl  sCltNCIH 
'  AV  s  '.  '  .  S  It  S  Y 
STANF  vNID,  vA  da  v.'-. 

\  IV.  Mat  iharan  Swowlnathan 

Laboratory  of  ^w#tt  io  and 
*  v  alviat  1  v'M  Reaea*  ch 
Sv-N'v'l  v't  Fducat  loti 

•>i iversity  v*t  '»**-*.v*  >etts 

Wsbe*-  ,  M  A  OlvV  > 

t  tV,  tk  ovl  8>mpsv'n 

v'f  t  ie*  ,'t  l\»t  a  At'.«ivsis  Reseat  c»* 
tdvkativ'nal  Testing  .Met  vice 
Pr  meets'-' ,  N.  O'  >A1 


Dr.  Klkumi  Tatsuoka 
Computer  Based  Education  Research 
LaPoratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbane,  IL  61801 

Dr.  Mjurlce  Tatsuoka 

Department  of  Educational  Psychology 

University  of  Illinois 

Champaign,  IL  618CI 

Or.  David  Thissen 
Department  of  Psychology 
University  of  Kansas 
Laurence.  KS  66C4i» 

Dr  .  Robert  Tsutakawa 
Dept,  of  Statistics 
University  of  PIssouri 
Columbia,  MO  65201 

Dr.  J.  Uhlaner 
Perceptronics ,  Inc. 

'■2/1  Varicl  Avenue 
Woodland  Hills,  CA  9136a 

Or,  Howard  Wainer 

Bureau  of  Social  Science  Research 

Hyh  *  street,  N,  a. 

DR.  THOMAS  aALLSTEH 
Payer  IETRIC  LABORATORY 
DAVIE  (ALL  113* 

UMVFP3ITY  'F  NORTH  CAROL 
APEL  HILL.  NC  2751R 

Or .  John  w *nnous 
Department  of  llanagement 
Plehtii  an  University 
l.us‘-  Lansing,  HI  aR62a 

Dr.  Phyllis  Weavtr 
Graduate  School  of  Education 
irvard  University 

Ljrsen  Hall,  Appian  Way 
nbridge.  PA  '213’ 

DR.  DUS Ah  E.  WHITFLY 
PSYCH’LOGY  DEPAHr-IENT 
’DIVER  TTY  OF  KANSAS 
LAWFr*:CE,  KANSAS  66nua 

Dr.  Wolfgang  Wildgruhe 
treitkrac  ftcamt 
Rosenberg  5300 
Bon.; ,  West  Germany  D-5  100 

Dr.  Robert  Woul 
D.  hool  Examination  Department 
'Diversity  of  London 
66-72  Dower  Street 

London  WCIE  6lE 
ENGLAND 

Dr.  Karl  Zinn 

Center  for  research  on  Learning 
dial  Teaching 
University  of  uichlgan 
Ann  Arbor,  HI  a.Rtna 


i 


Previous  Publications 


Proceedings  of  the  1077  Computerized  Adaptive  Testing  Conference,  Julv  1078. 

Serve  fs 

’0-1.  Relationships  among  Achievement  Level  Estimates  from  Three  Item  Characteristic  Curve 
Scoring  Methods,  .'pril  1070. 

Pinal  Report:  Btas-Eree  Computerized  Testing.  March  1070.  (NTIS  No.  AP  AC3b817b) 

’0-7.  Effects  of  Computerized  Adaptive  Testing  on  Black  and  White  Students.  March  1070. 

INTIS  No.  All  A0b 79283 

'0  1.  Computer  Programs  for  Scoring  Test  Pats  with  Item  Characteristic  Curve  Models.  February 
lO’N.  (NTIS  No.  AP  A0677523 

s  s.  An  Item  Bias  Investigation  of  a  Standardized  Aptitude  Test.  Peceraber  1078.  (NTIS  No. 

AP  AP64312) 

.  \  Construct  Validation  ot  Adaptive  Achievement  Testing.  November  1078. 

-  A  Comparison  ot  levels  and  Plmenslons  of  Performance  In  Black  and  White  Croups  on 

:  ^  t  s  ot  Vocabulary,  Mathematics,  and  Spatial  Ability.  October  1078.  (NTIS  No.  AP  AOb27973 
->  he  Effects  of  knowledge  of  Results  and  Test  Pifficulty  on  Ability  Test  Performance  and 

Psychological  Reactions  to  Testing.  September  1078. 

■>  l.  V  (.rnipartson  ot  the  Fairness  ot  Adaptive  and  Conventional  Testing  Strategies.  August 
10’8.  (NTIS  No.  AP  A0VHJb3 

An  Information  Comparison  ot  Conventional  and  Adaptive  Tests  in  the  Measurement  of 
Classroom  Achievement.  October  107’.  (NTIS  No.  AP  A047495) 
t- .  An  Adaptive  Testing  Strategy  for  Achievement  Test  Batteries.  October  1077.  (NTIS  No. 

AT  A046062) 

' Calibration  ot  an  Item  Pool  tot  the  Adaptive  Measurement  of  Achievement.  September  1077. 
t N IT S  No.  AP  A0448283 

•*.  l  Rapid  Item- Search  Procedure  for  Bayesian  Adaptive  Testing.  May  1077.  (NTIS  No. 

AP  A0410903 

*.  Accuracy  ot  Perceived  Test-Item  PI f f lcul t les .  Mav  1077.  (NTIS  No.  AP  A04 1084 3 

A  Comparison  ot  Information  Functions  ot  Multiple-Choice  and  E'ree- Response  Vocabulary 
Items.  April  1077. 

1.  Applications  of  Cornput er 1  zed  Adaptive  Testing.  March  1077.  (NTIS  No.  AP  A038114) 

Final  Report:  Computerized  Ability  Testing,  1072-1071.  April  107b.  (NTIS  No.  AP  A02411b3 
’b  1.  Fttects  ot  Item  Characterist  ics  on  Test  Fairness.  Peeemher  10'p.  INTIS  No.  AP  A0313933 
s.  Psvcholog  leal  Effects  of  Immediate  knowledge  of  Results  and  Adaptive  Abllttv  Testing, 
dune  10  >.  (NTIS  No.  AP  A027170) 

>->.  Effects  ot  Immediate  knowledge  of  Results  and  Adaptive  Testing  on  Ability  Test  Performance, 
dune  107b.  (NTIS  No.  AP  A0281473 

>  .  Effects  ot  Time  limits  on  Test-Taking  Behavior.  April  107b.  (NTIS  No.  AP  A0244223 

7b  1.  Some  Properties  ot  a  Bavesian  Adaptive  Ability  Testing  Strategy.  March  107b.  (NT'S  No. 

AP  A022964) 

?1-b.  A  Simulation  Studv  of  Stradaptlve  Ability  Testing.  Pecember  1071.  (NTIS  No.  AP  A0200bl) 
71-1.  Computerized  Adaptive  Trait  Measurement:  Problems  and  Prospects.  November  1071. 
tNTlS  No.  AP  A0 1867 5) 

71-s.  A  Studv  of  Computer-Administered  Stradaptlve  Ability  Testing.  October  107’>.  (NTIS  Nyy. 

AP  A0 187 181 

71-3.  Empirical  and  Simulation  Studies  ot  Vlexllevel  Abllttv  Testing.  Julv  1071.  (NTIS  No. 

AP  A0 1 31813 

71-2.  TFTRFST:  A  FORTRAN  IV  Program  for  Calculating  Tetrachorle  Correlations,  ''arch  1071. 

(NTIS  No.  AP  A007572) 

71-1.  An  Empirical  Comparison  ot  Two- St  age  and  Pvramldal  Adaptive  Ability  Testing.  February 
1071.  (NTIS  No.  All  AOOb733) . 

74-i.  Strategies  ot  Adaptive  Ability  Measurement.  Pecember  1074.  (NTIS  No.  AP  A004270) 

74-4.  Simulation  Studies  of  Two- St  age  Ability  Testing.  October  1074.  (NTIS  No.  AP  A001230) 

74-1.  An  Empirical  Investigation  of  Computer-Administered  Pvramldal  Ability  Testing.  Julv  1074. 
(NTIS  No.  AP  7811113 

'4-2.  A  Word  Knowledge  Item  Pool  tor  Adaptive  Ability  Measurement.  June  1074.  (NTIS  No. 

AP  7818043 

74-1.  A  Computer  Software  System  for  Adaptive  Abllttv  Measurement.  January  1074.  (NTIS  No. 

AP  7730M3 

71-3.  The  Stratified  Adaptive  Computerized  Abtlltv  Test.  September  1071.  (NTIS  No.  AP  7b8l'b3 
'1-2.  Ci<mpar  Ison  of  Four  Ifnp  1  it  ca  1  Item  Scoring  Procedures .  August  10’1. 

71-1.  Abllttv  Measurement :  Conventional  or  Adaptive?  February  1071.  (NTIS  No.  AP  7177883 

V'  * 1  .  *  * .  ?  ;*  *  *  '  ' '  <  '  '  ,%t  '  • '  ,•?,  '  J  *  ■  ’»■  \  '  •'  *  •  '  ?• 

"  ‘  \  j  *  t  \  M  .’  ’  *;  '  »*f*x  >  '  ,  ’ ,  •  •  . 

Copies  ot  these  reports  are  available,  while  supplies  last,  from: 

Psychometric  Methods  Program,  Pepartment  of  Psychology 
NbbO  Elliott  Hall,  Pntversltv  of  Minnesota 
71  East  River  Road,  Minneapolis,  Minnesota  11411 


